Exactly.  I'm no expert on maven either, but I like it's convenience for
classpath handling

Attached are my scripts.
- Hadoop-installer allows me to install different versions of hadoop
to local repo
- Pom has an assembly plugin (change mainClass and packageName to be
your target)
- Assembly does the packaging.  Run it with
 - mvn assembly:assembly -Dmaven.test.skip=true

The way I work is manage all dependencies in the pom, use "mvn
eclipse:eclipse" to keep eclipse buildpath correct.  Then I just run
everything in Eclipse with small input files until I am happy that it
works.  Then I build the jar with dependencies and copy it up to EC2
to run on the cluster.  Might not be the best way but seems fairly
efficient for me.

Cheers,

Tim


On Wed, Dec 3, 2008 at 10:42 PM, Scott Whitecross <[EMAIL PROTECTED]> wrote:
> Thanks Tim.
>
> We use Maven, though I'm not an expert on it.  Basically you are using Maven
> to take the dependencies, and package them in one large jar?  Basically
> unjar the contents of the jar and use those with your code I'm assuming?
>
>
> On Dec 3, 2008, at 9:25 AM, tim robertson wrote:
>
>> Can't answer your question exactly, but can let you know what I do.
>>
>> I build all dependencies into 1 jar, and by using Maven for my build
>> environment, when I assemble my jar, I am 100% sure all my
>> dependencies are collected together.  This is working very nicely for
>> me and I have used the same scripts for around 20 different jars that
>> I run on EC2 - each had different dependencies which would have been a
>> pain to manage seperately, but maven simplifies this massively.
>>
>> Let me know if you want any of my maven config for assembly etc if you
>> are a maven user...
>>
>> Cheers,
>>
>> Tim
>>
>>
>> On Wed, Dec 3, 2008 at 3:19 PM, Scott Whitecross <[EMAIL PROTECTED]> wrote:
>>>
>>> What's the best way to use third party libraries with Hadoop?  For
>>> example,
>>> I want to run a job with both a jar file containing the mob, and also
>>> extra
>>> libraries.  I noticed a couple solutions with a search, but I'm hoping
>>> for
>>> something better:
>>>
>>> - Merge the third party jar libraries into the job jar
>>> - Distribute the third party libraries across the cluster in the local
>>> boxes
>>> classpath.
>>>
>>> What I'd really like is a way to add an extra option to the hadoop jar
>>> command, such as hadoop/bin/hadoop jar myJar.jar myJobClass -classpath
>>> thirdpartyjar1.jar:jar2.jar:etc  args
>>>
>>> Anything exist like this?
>>>
>>
>
>
<assembly>
  <id>jar-with-dependencies</id>
  <formats>
    <format>jar</format>
  </formats>
  <includeBaseDirectory>false</includeBaseDirectory>
  <fileSets>
    <fileSet>
      <directory>target/classes</directory>
      <outputDirectory>/</outputDirectory>
    </fileSet>
  </fileSets>
  <dependencySets>
    <dependencySet>
      <outputDirectory>/</outputDirectory>
      <unpack>true</unpack>
      <scope>runtime</scope>
      <excludes>
        <!-- Do not include hadoop or log4j -->
        <exclude>org.apache.hadoop:hadoop-core</exclude>
        <exclude>log4j:log4j</exclude>
      </excludes>
    </dependencySet>
  </dependencySets>
</assembly>
<!-- 
	A little script to install hadoop.  It requires that hadoop-XXX-core.jar is in the same directory and run as:
	mvn -f hadoop-installer.xml install  -Dhadoop.version=0.19.0 -Dmaven.test.skip=true
-->
<project xmlns="http://maven.apache.org/POM/4.0.0"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd";>
  <modelVersion>4.0.0</modelVersion>
  <groupId>org.apache.hadoop</groupId>
  <artifactId>hadoop</artifactId>
  <name>Hadoop (${hadoop.version})</name>
  <packaging>jar</packaging>
  <version>${hadoop.version}</version>
  <build>
    <plugins>
      <plugin>
        <artifactId>maven-install-plugin</artifactId>
        <executions>
          <execution>
            <id>install-hadoop</id>
            <phase>install</phase>
            <goals>
              <goal>install-file</goal>
            </goals>
            <configuration>
              <file>hadoop-${hadoop.version}-core.jar</file>
              <groupId>org.apache.hadoop</groupId>
              <artifactId>hadoop-core</artifactId>
              <packaging>jar</packaging>
              <version>${hadoop.version}</version>
              <generatePom>true</generatePom>
              <createChecksum>true</createChecksum>
            </configuration>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>
</project>
<project xmlns="http://maven.apache.org/POM/4.0.0";
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
	xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd";>
	<modelVersion>4.0.0</modelVersion>
	<groupId>com.ibiodiversity.index</groupId>
	<artifactId>index-hadoop</artifactId>
	<packaging>jar</packaging>
	<version>1.0-SNAPSHOT</version>
	<name>hadoop-server</name>
	<url>http://maven.apache.org</url>
	<repositories>
		<repository>
			<id>geotools</id>
			<url>http://maven.geotools.fr/repository</url>
		</repository>
		<repository>
			<id>central</id>
			<url>http://repo1.maven.org/maven2</url>
		</repository>
		<repository>
			<id>appfuse</id>
			<url>http://static.appfuse.org/repository</url>
		</repository>
    <repository> 
      <name>jets3t</name> 
      <id>jets3t</id> 
      <url>http://jets3t.s3.amazonaws.com/maven2</url> 
    </repository> 		
	</repositories>
	<dependencies>
		<dependency>
			<groupId>junit</groupId>
			<artifactId>junit</artifactId>
			<version>${junit.version}</version>
			<scope>test</scope>
		</dependency>
		<dependency>
			<groupId>commons-logging</groupId>
			<artifactId>commons-logging</artifactId>
			<version>${commons-logging.version}</version>
		</dependency>
		<dependency>
			<groupId>commons-cli</groupId>
			<artifactId>commons-cli</artifactId>
			<version>${commons-cli.version}</version>
		</dependency>
		<dependency>
			<groupId>commons-httpclient</groupId>
			<artifactId>commons-httpclient</artifactId>
			<version>${commons-httpclient.version}</version>
		</dependency>
		<dependency>
			<groupId>log4j</groupId>
			<artifactId>log4j</artifactId>
			<version>${log4j.version}</version>
		</dependency>
		<dependency>
			<groupId>org.apache.hadoop</groupId>
			<artifactId>hadoop-core</artifactId>
			<version>${hadoop.version}</version>
		</dependency>
		<dependency>
			<groupId>org.geotools</groupId>
			<artifactId>gt2-main</artifactId>
			<version>${gt2.version}</version>
		</dependency>
		<dependency>
			<groupId>org.geotools</groupId>
			<artifactId>gt2-shapefile</artifactId>
			<version>${gt2.version}</version>
		</dependency>
		<dependency>
			<groupId>org.geotools</groupId>
			<artifactId>gt2-epsg-wkt</artifactId>
			<version>${gt2.version}</version>
		</dependency>
		<dependency>
			<groupId>com.vividsolutions</groupId>
			<artifactId>jts</artifactId>
			<version>${jts.version}</version>
		</dependency>
    <dependency>
      <groupId>commons-math</groupId>
      <artifactId>commons-math</artifactId>
      <version>1.1</version>
    </dependency>		
		
<dependency> 
      <groupId>net.java.dev.jets3t</groupId> 
      <artifactId>jets3t</artifactId> 
      <version>0.6.0</version> 
    </dependency> 		
	</dependencies>
  <build>
	<plugins>
			<!-- Use 1.6 for Hadoop 0.19.0 otherwise, 1.5 will suffice -->
			<plugin>
				<artifactId>maven-compiler-plugin</artifactId>
				<version>2.0.2</version>
				<configuration>
					<source>1.6</source>
					<target>1.6</target>
				</configuration>
			</plugin>
		
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-assembly-plugin</artifactId>
        <configuration>
					<archive>
					  <manifest>
						<mainClass>com.ibiodiversity.index.mapreduce.occurrence.geometry.point2.PolygonOccurrenceIntersection</mainClass>
						<packageName>com.ibiodiversity.index.mapreduce.occurrence.geometry.point2</packageName>
					  </manifest>
					</archive>
          <descriptors>
            <descriptor>assemble.xml</descriptor>
          </descriptors>
        </configuration>
      </plugin>
    </plugins>
  </build>
	<properties>
		<!-- Framework dependency versions -->
		<commons-logging.version>1.0.4</commons-logging.version>
		<commons-cli.version>2.0-SNAPSHOT</commons-cli.version>
		<log4j.version>1.2.13</log4j.version>
		<hadoop.version>0.19.0</hadoop.version>
		<jts.version>1.8</jts.version>
		<commons-httpclient.version>3.1</commons-httpclient.version>
		<gt2.version>2.4.5</gt2.version>
		
		<!-- Testing dependency versions -->
		<junit.version>4.4</junit.version>
		
	</properties>
</project>

Reply via email to