Re: How to compile Spark with private build of Hadoop

[Steve Loughran]

On 8 Mar 2016, at 07:23, Lu, Yingqi 
> wrote:

Thank you for the quick reply. I am very new to maven and always use the 
default settings. Can you please be a little more specific on the instructions?

I think all the jar files from Hadoop build are located at 
Hadoop-3.0.0-SNAPSHOT/share/hadoop. Which ones I need to use to compile Spark 
and how can I change the pom.xml?

Thanks,
Lucy





It's simple to do this locally; no need for a remote server

You do need to do every morning, and do not try to run a build over midnight, 
as that confuses Maven. Just bear in mind "the first build that maven does 
every day, it will try to get snapshots remotely, if they aren't local"

1. in hadoop-trunk:
mvn install -DskipTests

This will publish the 3.0.0-SNAPSHOT JARs into ~/.m2/repository , where they 
will be picked up by dependent builds for the test of the day

2. in spark

mvn install -DskipTests -Phadoop-2.6 -Dhadoop.version=3.0.0-SNAPSHOT

That turns on the Hadoop 2.6+ profile, but sets the Hadoop version to build to 
be the 3.0.0 one you built in step (1)

3. Go and have a coffee; wait for the spark build to finish

That's all you need to do to get a version of spark built with your hadoop 
version.

It may be that spark fails to compile against Hadoop 2.9.0-SNAPSHOT or 
3.0.0-SNAPSHOT. If that happens, consider it a regression in Hadoop, file a bug 
there. I've been working with Hadoop 2.8.0-SNAPSHOT without problems, except 
for where, in the split of HDFS in to client and server JARs/POMs 
(hadoop-hdfs-client and hadoop-hdfs), the client JAR had left out some classes 
I expected to be available. That's been fixed, but don't be afraid to complain 
yourself if you find a problem: it's in the nightly build phase where 
regressions can be fixed within 24h

Re: RE: How to compile Spark with private build of Hadoop

[fightf...@163.com]

Hi, there,
You may try to use nexus to establish maven local repository. I think this link 
would be helpful. 
http://www.sonatype.org/nexus/2015/02/27/setup-local-nexus-repository-and-deploy-war-file-from-maven/
 

After you had done the repository, you may use maven-deploy-plugin to deploy 
your customized 
hadoop jar and relative pom.xml to nexus repository. Check the link for 
reference: 
https://books.sonatype.com/nexus-book/reference/staging-deployment.html 




fightf...@163.com
 
From: Lu, Yingqi
Date: 2016-03-08 15:23
To: fightf...@163.com; user
Subject: RE: How to compile Spark with private build of Hadoop
Thank you for the quick reply. I am very new to maven and always use the 
default settings. Can you please be a little more specific on the instructions?
 
I think all the jar files from Hadoop build are located at 
Hadoop-3.0.0-SNAPSHOT/share/hadoop. Which ones I need to use to compile Spark 
and how can I change the pom.xml?
 
Thanks,
Lucy
 
From: fightf...@163.com [mailto:fightf...@163.com] 
Sent: Monday, March 07, 2016 11:15 PM
To: Lu, Yingqi ; user 
Subject: Re: How to compile Spark with private build of Hadoop
 
I think you can establish your own maven repository and deploy your modified 
hadoop binary jar 
with your modified version number. Then you can add your repository in spark 
pom.xml and use 
mvn -Dhadoop.version=
 


fightf...@163.com
 
From: Lu, Yingqi
Date: 2016-03-08 15:09
To: user@spark.apache.org
Subject: How to compile Spark with private build of Hadoop
Hi All,
 
I am new to Spark and I have a question regarding to compile Spark. I modified 
trunk version of Hadoop source code. How can I compile Spark (standalone mode) 
with my modified version of Hadoop (HDFS, Hadoop-common and etc.)?
 
Thanks a lot for your help!
 
Thanks,
Lucy
 
 
 
 

Re: How to compile Spark with private build of Hadoop

[Saisai Shao]

I think the first step is to publish your in-house built Hadoop related
jars to your local maven or ivy repo, and then change the Spark building
profiles like -Phadoop-2.x (you could use 2.7 or you have to change the pom
file if you met jar conflicts) -Dhadoop.version=3.0.0-SNAPSHOT to build
against your specified version.

Thanks
Saisai

RE: How to compile Spark with private build of Hadoop

[Lu, Yingqi]

Thank you for the quick reply. I am very new to maven and always use the 
default settings. Can you please be a little more specific on the instructions?

I think all the jar files from Hadoop build are located at 
Hadoop-3.0.0-SNAPSHOT/share/hadoop. Which ones I need to use to compile Spark 
and how can I change the pom.xml?

Thanks,
Lucy

From: fightf...@163.com [mailto:fightf...@163.com]
Sent: Monday, March 07, 2016 11:15 PM
To: Lu, Yingqi ; user 
Subject: Re: How to compile Spark with private build of Hadoop

I think you can establish your own maven repository and deploy your modified 
hadoop binary jar
with your modified version number. Then you can add your repository in spark 
pom.xml and use
mvn -Dhadoop.version=


fightf...@163.com

From: Lu, Yingqi
Date: 2016-03-08 15:09
To: user@spark.apache.org
Subject: How to compile Spark with private build of Hadoop
Hi All,

I am new to Spark and I have a question regarding to compile Spark. I modified 
trunk version of Hadoop source code. How can I compile Spark (standalone mode) 
with my modified version of Hadoop (HDFS, Hadoop-common and etc.)?

Thanks a lot for your help!

Thanks,
Lucy

Re: How to compile Spark with private build of Hadoop

[fightf...@163.com]

I think you can establish your own maven repository and deploy your modified 
hadoop binary jar 
with your modified version number. Then you can add your repository in spark 
pom.xml and use 
mvn -Dhadoop.version=



fightf...@163.com
 
From: Lu, Yingqi
Date: 2016-03-08 15:09
To: user@spark.apache.org
Subject: How to compile Spark with private build of Hadoop
Hi All,
 
I am new to Spark and I have a question regarding to compile Spark. I modified 
trunk version of Hadoop source code. How can I compile Spark (standalone mode) 
with my modified version of Hadoop (HDFS, Hadoop-common and etc.)?
 
Thanks a lot for your help!
 
Thanks,
Lucy