first setup the build environment. I use a script to setup following properties, you need to modify it: export M2_HOME=~/development/apache-maven-3.0.4 export M2=$M2_HOME/bin export ANT_HOME=/opt/apache-ant-1.8.4 export PATH=$M2:$ANT_HOME/bin:$PATH: export GREP_OPTIONS='--color=auto' export HADOOP_HOME=~/development/hadoop-1.1.2 export HADOOP_LOG_DIR=$HADOOP_HOME/logs export ECLIPSE_HOME=~/development/eclipse export FORREST_HOME=~/development/apache-forrest-0.8 export PATH=$PATH:$FORREST_HOME/bin export JAVA_HOME=~/development/ibm-java-x86_64-60 export PATH= $JAVA_HOME/bin:$PATH:$M2:$ANT_HOME/bin:~/development/protobuf-2.4.1/src:/usr/lib64
export CLASSPATH=$ANT_HOME/lib:$CLASSPATH:$JAVA_HOME/jre/lib:$ANT_HOME/lib: export LD_LIBRARY_PATH= $HADOOP_HOME/build/native/Linux-amd64-64/lib:$ANT_HOME/lib:/usr/lib64 umask 0022 then build by: ant clean package -Djava5.home=/usr/java/jdk1.5.0_22 -Dforrest.home=/home/sanjar/development/apache-forrest-0.8 -Dcompile.native=true Best Regards Amir Sanjar System Management Architect PowerLinux Open Source Hadoop development lead IBM Senior Software Engineer Phone# 512-286-8393 Fax# 512-838-8858 From: Mohammad Mustaqeem <3m.mustaq...@gmail.com> To: common-dev <common-dev@hadoop.apache.org>, Date: 04/09/2013 08:40 AM Subject: Re: Building Hadoop from source code @Amir, how to compile this?? The process will be same as using "mvn package -Pdist -Dtar -DskipTests" or something else?? On Tue, Apr 9, 2013 at 7:00 PM, Amir Sanjar <v1san...@us.ibm.com> wrote: > try hadoop 1.1.2.. > > Best Regards > Amir Sanjar > > System Management Architect > PowerLinux Open Source Hadoop development lead > IBM Senior Software Engineer > Phone# 512-286-8393 > Fax# 512-838-8858 > > > > [image: Inactive hide details for Mohammad Mustaqeem ---04/09/2013 > 08:18:02 AM---@Ling, i am very new. I was about to ask you the same.]Mohammad > Mustaqeem ---04/09/2013 08:18:02 AM---@Ling, i am very new. I was about > to ask you the same. > > From: Mohammad Mustaqeem <3m.mustaq...@gmail.com> > To: common-dev <common-dev@hadoop.apache.org>, > Date: 04/09/2013 08:18 AM > Subject: Re: Building Hadoop from source code > ------------------------------ > > > > @Ling, i am very new. > I was about to ask you the same. > Which version is easy to handle? > I want to change something in HDFS. > Please, suggest. > > > On Tue, Apr 9, 2013 at 6:29 PM, Ling Kun <lkun.e...@gmail.com> wrote: > > > @Mohammad > > > > Sorry, since the internet connection rate I am using is tooooooo low, > It > > takes me a looooong time to download the tar file you showed above. > > > > however, here is some of my opinion based on my local develop env, > and i > > am also a fresh hadooper here. > > > > 1. what you have download from the trunk is Hadoop 3.0, which you can > > see from the tar.gz filename. While the link above is about Hadoop 1.0, > > and 2.0. So it seems hadoop developers have adjust the distributed tar > > file. > > > > 2. When looking into the bin/hadoop script in both tar.gz file. I > think > > that you will find out that most of them are the same. This file will > make > > a MR job start. > > > > 3. What version of hadoop do you plan to work on? 2.0 or 3.0? > > > > Ling Kun > > > > > > On Tue, Apr 9, 2013 at 8:25 PM, Mohammad Mustaqeem > > <3m.mustaq...@gmail.com>wrote: > > > > > @Ling, Is it the same file as we download from > > > http://apache.techartifact.com/mirror/hadoop/common/ ?? > > > But the tree structure of the build tar file is not same as downloaded > > from > > > the site. > > > > > > @Ling plz guide me. I have been stuck at this state from 1 month . > > > I didn't find link that explain in detail. The material that I found is > > > very old that uses ant. > > > I want to just build hadoop from source code. > > > Plz, tell > > > > > > > > > On Tue, Apr 9, 2013 at 5:37 PM, Ling Kun <lkun.e...@gmail.com> wrote: > > > > > > > @Mohammad: > > > > 1. have you successfully run the command Steve suggested? "mvn > > package > > > > -Pdist -Dtar -DskipTests -Dmaven.javadoc.skip=true" > > > > 2. If yes, and then you can use the command " find . -name "*tar*" > " > > to > > > > find the tar file, which is in the path > > > > "./hadoop-dist/target/hadoop-3.0.0-SNAPSHOT.tar.gz" > > > > > > > > 3. Sorry, I make a mistake about the tar file name, It is tar.gz > > > > > > > > > > > > @ maisnam ns: > > > > I am afraid the version you are using is not the same as Mohammad > > > used. > > > > > > > > > > > > Ling Kun > > > > > > > > > > > > On Tue, Apr 9, 2013 at 7:58 PM, Mohammad Mustaqeem > > > > <3m.mustaq...@gmail.com>wrote: > > > > > > > > > @maisnam, I am using ubuntu and after running svn I get hadoop > folder > > > as > > > > > given in screenshot. > > > > > I didn't finf any build folder > > > > > > > > > > > > > > > On Tue, Apr 9, 2013 at 5:22 PM, maisnam ns <maisnam...@gmail.com> > > > wrote: > > > > > > > > > >> @@Mohammad Mustaqeem please ignore the earlier mail as the > > screenshots > > > > >> where not copied in the mail I have attached a doc file. > > > > >> > > > > >> You can follow the steps earlier using the screenshots I have > > > attached. > > > > >> > > > > >> > > > > >> On Tue, Apr 9, 2013 at 5:15 PM, maisnam ns <maisnam...@gmail.com> > > > > wrote: > > > > >> > > > > >>> @Mohammad Mustaqeem please follow the screenshots one by one > > > > >>> > > > > >>> > > > > >>> cd build you will find > > > > >>> > > > > >>> > > > > >>> Now cd hadoop-0.20.3-dev you will find the below screenshot > > > > >>> > > > > >>> > > > > >>> Now the bin contains all the scripts you may need to start hadoop > > > > >>> 1. Go to that bin directory and format your namenode > > > > >>> 2. configure you site-xmls > > > > >>> 3. start-all.sh note there are other ways also > > > > >>> 4. do jps to see if all daemons are running > > > > >>> > > > > >>> > > > > >>> > > > > >>> On Tue, Apr 9, 2013 at 4:54 PM, Mohammad Mustaqeem < > > > > >>> 3m.mustaq...@gmail.com> wrote: > > > > >>> > > > > >>>> Please, anyone tell me where is the build hadoop?? > > > > >>>> that can be used to install the hadoop cluster. > > > > >>>> > > > > >>>> > > > > >>>> On Tue, Apr 9, 2013 at 2:32 PM, Mohammad Mustaqeem > > > > >>>> <3m.mustaq...@gmail.com>wrote: > > > > >>>> > > > > >>>> > @Ling, you mean to say to run "find -name *tar*"?? > > > > >>>> > I run but don't know which file will be used to install > Hadoop. > > > > >>>> > > > > > >>>> > @Niranjan > > > > >>>> > I haven't changed anything I just executed "svn checkout > > > > >>>> > > > http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-trunk"; > > > > >>>> to get > > > > >>>> > the Hadoop source code in the "hadoop-trunk" directory. > > > > >>>> > After that I executed the "cd hadoop-trunk" and finally > executed > > > > "mvn > > > > >>>> > package -Pdist -Dtar -DskipTests -Dmaven.javadoc.skip=true". > > > > >>>> > I want to ask what is location of tar file created which can > be > > > used > > > > >>>> to > > > > >>>> > install the Hadoop ?? > > > > >>>> > > > > > >>>> > > > > > >>>> > On Tue, Apr 9, 2013 at 2:08 PM, maisnam ns < > > maisnam...@gmail.com> > > > > >>>> wrote: > > > > >>>> > > > > > >>>> >> @Mohammad Mustaqeem - Once you create the patch as given in > the > > > > 'How > > > > >>>> to > > > > >>>> >> contribute to Hadoop' and apply the patch. The changes will > be > > > > >>>> reflected > > > > >>>> >> in the files you have modified. > > > > >>>> >> > > > > >>>> >> 1.Now you trigger the build ant or maven. I tried with ant so > > the > > > > ant > > > > >>>> >> command I gave is ant clean compile bin-package. Don't > forget > > to > > > > >>>> download > > > > >>>> >> ivy.jar and copy into you ant home/ lib folder. Once the > build > > is > > > > >>>> >> triggered > > > > >>>> >> Hadoop should get built along with the changes you made. > > > > >>>> >> > > > > >>>> >> If , I am not mistaken , you modified some hadoop files say > > > > >>>> >> BlockLocation.java, in your > > > > >>>> >> Hadoopx.x\src\core\org\apache\hadoop\fs\BlockLocation.java. > > > > >>>> >> > > > > >>>> >> The jar will be in > > Hadoopx.x\build\hadoop-0.20.3-dev-core.jar(In > > > my > > > > >>>> >> version) > > > > >>>> >> > > > > >>>> >> Hope this clears your doubt. > > > > >>>> >> > > > > >>>> >> Regards > > > > >>>> >> Niranjan Singh > > > > >>>> >> > > > > >>>> >> > > > > >>>> >> On Tue, Apr 9, 2013 at 1:38 PM, Mohammad Mustaqeem > > > > >>>> >> <3m.mustaq...@gmail.com>wrote: > > > > >>>> >> > > > > >>>> >> > @Steve > > > > >>>> >> > I am new to Hadoop developement. > > > > >>>> >> > Can you please tell me, whats is the location of tar file?? > > > > >>>> >> > > > > > >>>> >> > > > > > >>>> >> > On Tue, Apr 9, 2013 at 12:09 AM, Steve Loughran < > > > > >>>> ste...@hortonworks.com > > > > >>>> >> > >wrote: > > > > >>>> >> > > > > > >>>> >> > > On 8 April 2013 16:08, Mohammad Mustaqeem < > > > > >>>> 3m.mustaq...@gmail.com> > > > > >>>> >> > wrote: > > > > >>>> >> > > > > > > >>>> >> > > > Please, tell what I am doing wrong?? > > > > >>>> >> > > > Whats the problem?? > > > > >>>> >> > > > > > > > >>>> >> > > > > > > >>>> >> > > a lot of these seem to be network-related tests. You can > > turn > > > > >>>> off all > > > > >>>> >> the > > > > >>>> >> > > tests; look in BUILDING.TXT at the root of the source > tree > > > for > > > > >>>> the > > > > >>>> >> > various > > > > >>>> >> > > operations, then add -DskipTests to the end of every > > command, > > > > >>>> such as > > > > >>>> >> > > > > > > >>>> >> > > mvn package -Pdist -Dtar -DskipTests > > > > >>>> >> > > > > > > >>>> >> > > to build the .tar packages > > > > >>>> >> > > > > > > >>>> >> > > mvn package -Pdist -Dtar -DskipTests > > > -Dmaven.javadoc.skip=true > > > > >>>> >> > > to turn off the javadoc creation too, for an even faster > > > build > > > > >>>> >> > > > > > > >>>> >> > > > > > >>>> >> > > > > > >>>> >> > > > > > >>>> >> > -- > > > > >>>> >> > *With regards ---* > > > > >>>> >> > *Mohammad Mustaqeem*, > > > > >>>> >> > M.Tech (CSE) > > > > >>>> >> > MNNIT Allahabad > > > > >>>> >> > 9026604270 > > > > >>>> >> > > > > > >>>> >> > > > > >>>> > > > > > >>>> > > > > > >>>> > > > > > >>>> > -- > > > > >>>> > *With regards ---* > > > > >>>> > *Mohammad Mustaqeem*, > > > > >>>> > M.Tech (CSE) > > > > >>>> > MNNIT Allahabad > > > > >>>> > 9026604270 > > > > >>>> > > > > > >>>> > > > > > >>>> > > > > > >>>> > > > > >>>> > > > > >>>> -- > > > > >>>> *With regards ---* > > > > >>>> *Mohammad Mustaqeem*, > > > > >>>> M.Tech (CSE) > > > > >>>> MNNIT Allahabad > > > > >>>> 9026604270 > > > > >>>> > > > > >>> > > > > >>> > > > > >> > > > > > > > > > > > > > > > -- > > > > > *With regards ---* > > > > > *Mohammad Mustaqeem*, > > > > > M.Tech (CSE) > > > > > MNNIT Allahabad > > > > > 9026604270 > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > http://www.lingcc.com > > > > > > > > > > > > > > > > -- > > > *With regards ---* > > > *Mohammad Mustaqeem*, > > > M.Tech (CSE) > > > MNNIT Allahabad > > > 9026604270 > > > > > > > > > > > -- > > http://www.lingcc.com > > > > > > -- > *With regards ---* > *Mohammad Mustaqeem*, > M.Tech (CSE) > MNNIT Allahabad > 9026604270 > > -- *With regards ---* *Mohammad Mustaqeem*, M.Tech (CSE) MNNIT Allahabad 9026604270
