I used the following command: make-distribution.sh --name custom-spark --tgz -Phadoop-2.4 -Phive -Phive-thriftserver -Pyarn
spark-1.6.0-SNAPSHOT-bin-custom-spark.tgz was generated (with patch from SPARK-11348) Can you try above command ? Thanks On Tue, Oct 27, 2015 at 7:03 AM, Kayode Odeyemi <drey...@gmail.com> wrote: > Ted, I switched to this: > > ./make-distribution.sh --name spark-latest --tgz -Dhadoop.version=2.6.0 > -Phadoop-2.6 -Phive -Phive-thriftserver -Pyarn -DskipTests clean package -U > > Same error. No .gz file. Here's the bottom output log: > > + rm -rf /home/emperor/javaprojects/spark/dist > + mkdir -p /home/emperor/javaprojects/spark/dist/lib > + echo 'Spark [WARNING] See > http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin (git revision > 3689beb) built for Hadoop [WARNING] See > http://docs.codehaus.org/display/MAVENUSER/Shade+Pl > + echo 'Build flags: -Dhadoop.version=2.6.0' -Phadoop-2.6 -Phive > -Phive-thriftserver -Pyarn -DskipTests clean package -U > + cp > /home/emperor/javaprojects/spark/assembly/target/scala-2.10/spark-assembly-1.6.0-SNAPSHOT-hadoop2.6.0.jar > /home/emperor/javaprojects/spark/dist/lib/ > + cp > /home/emperor/javaprojects/spark/examples/target/scala-2.10/spark-examples-1.6.0-SNAPSHOT-hadoop2.6.0.jar > /home/emperor/javaprojects/spark/dist/lib/ > + cp > /home/emperor/javaprojects/spark/network/yarn/target/scala-2.10/spark-1.6.0-SNAPSHOT-yarn-shuffle.jar > /home/emperor/javaprojects/spark/dist/lib/ > + mkdir -p /home/emperor/javaprojects/spark/dist/examples/src/main > + cp -r /home/emperor/javaprojects/spark/examples/src/main > /home/emperor/javaprojects/spark/dist/examples/src/ > + '[' 1 == 1 ']' > + cp > /home/emperor/javaprojects/spark/lib_managed/jars/datanucleus-api-jdo-3.2.6.jar > /home/emperor/javaprojects/spark/lib_managed/jars/datanucleus-core-3.2.10.jar > /home/emperor/javaprojects > ed/jars/datanucleus-rdbms-3.2.9.jar > /home/emperor/javaprojects/spark/dist/lib/ > + cp /home/emperor/javaprojects/spark/LICENSE > /home/emperor/javaprojects/spark/dist > + cp -r /home/emperor/javaprojects/spark/licenses > /home/emperor/javaprojects/spark/dist > + cp /home/emperor/javaprojects/spark/NOTICE > /home/emperor/javaprojects/spark/dist > + '[' -e /home/emperor/javaprojects/spark/CHANGES.txt ']' > + cp -r /home/emperor/javaprojects/spark/data > /home/emperor/javaprojects/spark/dist > + mkdir /home/emperor/javaprojects/spark/dist/conf > + cp /home/emperor/javaprojects/spark/conf/docker.properties.template > /home/emperor/javaprojects/spark/conf/fairscheduler.xml.template > /home/emperor/javaprojects/spark/conf/log4j.properties > emperor/javaprojects/spark/conf/metrics.properties.template > /home/emperor/javaprojects/spark/conf/slaves.template > /home/emperor/javaprojects/spark/conf/spark-defaults.conf.template /home/em > ts/spark/conf/spark-env.sh.template > /home/emperor/javaprojects/spark/dist/conf > + cp /home/emperor/javaprojects/spark/README.md > /home/emperor/javaprojects/spark/dist > + cp -r /home/emperor/javaprojects/spark/bin > /home/emperor/javaprojects/spark/dist > + cp -r /home/emperor/javaprojects/spark/python > /home/emperor/javaprojects/spark/dist > + cp -r /home/emperor/javaprojects/spark/sbin > /home/emperor/javaprojects/spark/dist > + cp -r /home/emperor/javaprojects/spark/ec2 > /home/emperor/javaprojects/spark/dist > + '[' -d /home/emperor/javaprojects/spark/R/lib/SparkR ']' > + '[' false == true ']' > + '[' true == true ']' > + TARDIR_NAME='spark-[WARNING] See > http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin-bin-spark-latest' > + TARDIR='/home/emperor/javaprojects/spark/spark-[WARNING] See > http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin-bin-spark-latest' > + rm -rf '/home/emperor/javaprojects/spark/spark-[WARNING] See > http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin-bin-spark-latest' > + cp -r /home/emperor/javaprojects/spark/dist > '/home/emperor/javaprojects/spark/spark-[WARNING] See > http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin-bin-spark-latest' > cp: cannot create directory > `/home/emperor/javaprojects/spark/spark-[WARNING] See > http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin-bin-spark-latest': > No such file or directory > > > On Tue, Oct 27, 2015 at 2:14 PM, Ted Yu <yuzhih...@gmail.com> wrote: > >> Can you try the same command shown in the pull request ? >> >> Thanks >> >> On Oct 27, 2015, at 12:40 AM, Kayode Odeyemi <drey...@gmail.com> wrote: >> >> Thank you. >> >> But I'm getting same warnings and it's still preventing the archive from >> being generated. >> >> I've ran this on both OSX Lion and Ubuntu 12. Same error. No .gz file >> >> On Mon, Oct 26, 2015 at 9:10 PM, Ted Yu <yuzhih...@gmail.com> wrote: >> >>> Looks like '-Pyarn' was missing in your command. >>> >>> On Mon, Oct 26, 2015 at 12:06 PM, Kayode Odeyemi <drey...@gmail.com> >>> wrote: >>> >>>> I used this command which is synonymous to what you have: >>>> >>>> ./make-distribution.sh --name spark-latest --tgz --mvn mvn >>>> -Dhadoop.version=2.6.0 -Phadoop-2.6 -Phive -Phive-thriftserver -DskipTests >>>> clean package -U >>>> >>>> But I still see WARNINGS like this in the output and no .gz file >>>> created: >>>> >>>> cp: /usr/local/spark-latest/spark-[WARNING] See >>>> http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin-bin-spark-latest/python/test_support/sql/parquet_partitioned/year=2015/month=10/day=26/.part-r-00005.gz.parquet.crc: >>>> No such file or directory >>>> cp: /usr/local/spark-latest/spark-[WARNING] See >>>> http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin-bin-spark-latest/python/test_support/sql/parquet_partitioned/year=2015/month=10/day=26/part-r-00005.gz.parquet: >>>> No such file or directory >>>> cp: /usr/local/spark-latest/spark-[WARNING] See >>>> http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin-bin-spark-latest/python/test_support/sql/parquet_partitioned/year=2015/month=9: >>>> No such file or directory >>>> cp: >>>> /usr/local/spark-latest/dist/python/test_support/sql/parquet_partitioned/year=2015/month=9: >>>> unable to copy extended attributes to >>>> /usr/local/spark-latest/spark-[WARNING] See >>>> http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin-bin-spark-latest/python/test_support/sql/parquet_partitioned/year=2015/month=9: >>>> No such file or directory >>>> cp: /usr/local/spark-latest/spark-[WARNING] See >>>> http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin-bin-spark-latest/python/test_support/sql/parquet_partitioned/year=2015/month=9/day=1: >>>> No such file or directory >>>> cp: >>>> /usr/local/spark-latest/dist/python/test_support/sql/parquet_partitioned/year=2015/month=9/day=1: >>>> unable to copy extended attributes to >>>> /usr/local/spark-latest/spark-[WARNING] See >>>> http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin-bin-spark-latest/python/test_support/sql/parquet_partitioned/year=2015/month=9/day=1: >>>> No such file or directory >>>> cp: /usr/local/spark-latest/spark-[WARNING] See >>>> http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin-bin-spark-latest/python/test_support/sql/parquet_partitioned/year=2015/month=9/day=1/.part-r-00007.gz.parquet.crc: >>>> No such file or directory >>>> >>>> On Mon, Oct 26, 2015 at 8:58 PM, Ted Yu <yuzhih...@gmail.com> wrote: >>>> >>>>> If you use the command shown in: >>>>> https://github.com/apache/spark/pull/9281 >>>>> >>>>> You should have got the following: >>>>> >>>>> >>>>> ./dist/python/test_support/sql/parquet_partitioned/year=2014/month=9/day=1/part-r-00008.gz.parquet >>>>> >>>>> ./dist/python/test_support/sql/parquet_partitioned/year=2015/month=9/day=1/part-r-00007.gz.parquet >>>>> >>>>> ./dist/python/test_support/sql/parquet_partitioned/year=2015/month=10/day=25/part-r-00004.gz.parquet >>>>> >>>>> ./dist/python/test_support/sql/parquet_partitioned/year=2015/month=10/day=25/part-r-00002.gz.parquet >>>>> >>>>> ./dist/python/test_support/sql/parquet_partitioned/year=2015/month=10/day=26/part-r-00005.gz.parquet >>>>> >>>>> On Mon, Oct 26, 2015 at 11:47 AM, Kayode Odeyemi <drey...@gmail.com> >>>>> wrote: >>>>> >>>>>> I see a lot of stuffs like this after the a successful maven build: >>>>>> >>>>>> cp: /usr/local/spark-latest/spark-[WARNING] See >>>>>> http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin-bin-spark-latest/python/test_support/sql/parquet_partitioned/year=2014/month=9/day=1/ >>>>>> part-r-00008.gz.parquet: No such file or directory >>>>>> >>>>>> Seems it fails when it tries to package the build as an archive. >>>>>> >>>>>> I'm using the latest code on github master. >>>>>> >>>>>> Any ideas please? >>>>>> >>>>>> On Mon, Oct 26, 2015 at 6:20 PM, Yana Kadiyska < >>>>>> yana.kadiy...@gmail.com> wrote: >>>>>> >>>>>>> In 1.4 ./make_distribution produces a .tgz file in the root >>>>>>> directory (same directory that make_distribution is in) >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Mon, Oct 26, 2015 at 8:46 AM, Kayode Odeyemi <drey...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> The ./make_distribution task completed. However, I can't seem to >>>>>>>> locate the >>>>>>>> .tar.gz file. >>>>>>>> >>>>>>>> Where does Spark save this? or should I just work with the dist >>>>>>>> directory? >>>>>>>> >>>>>>>> On Fri, Oct 23, 2015 at 4:23 PM, Kayode Odeyemi <drey...@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> I saw this when I tested manually (without ./make-distribution) >>>>>>>>> >>>>>>>>> Detected Maven Version: 3.2.2 is not in the allowed range 3.3.3. >>>>>>>>> >>>>>>>>> So I simply upgraded maven to 3.3.3. >>>>>>>>> >>>>>>>>> Resolved. Thanks >>>>>>>>> >>>>>>>>> On Fri, Oct 23, 2015 at 3:17 PM, Sean Owen <so...@cloudera.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> This doesn't show the actual error output from Maven. I have a >>>>>>>>>> strong >>>>>>>>>> guess that you haven't set MAVEN_OPTS to increase the memory >>>>>>>>>> Maven can >>>>>>>>>> use. >>>>>>>>>> >>>>>>>>>> On Fri, Oct 23, 2015 at 6:14 AM, Kayode Odeyemi < >>>>>>>>>> drey...@gmail.com> wrote: >>>>>>>>>> > Hi, >>>>>>>>>> > >>>>>>>>>> > I can't seem to get a successful maven build. Please see >>>>>>>>>> command output >>>>>>>>>> > below: >>>>>>>>>> > >>>>>>>>>> > bash-3.2$ ./make-distribution.sh --name spark-latest --tgz >>>>>>>>>> --mvn mvn >>>>>>>>>> > -Dhadoop.version=2.7.0 -Phadoop-2.7 -Phive -Phive-thriftserver >>>>>>>>>> -DskipTests >>>>>>>>>> > clean package >>>>>>>>>> > +++ dirname ./make-distribution.sh >>>>>>>>>> > ++ cd . >>>>>>>>>> > ++ pwd >>>>>>>>>> > + SPARK_HOME=/usr/local/spark-latest >>>>>>>>>> > + DISTDIR=/usr/local/spark-latest/dist >>>>>>>>>> > + SPARK_TACHYON=false >>>>>>>>>> > + TACHYON_VERSION=0.7.1 >>>>>>>>>> > + TACHYON_TGZ=tachyon-0.7.1-bin.tar.gz >>>>>>>>>> > + >>>>>>>>>> > TACHYON_URL= >>>>>>>>>> https://github.com/amplab/tachyon/releases/download/v0.7.1/tachyon-0.7.1-bin.tar.gz >>>>>>>>>> > + MAKE_TGZ=false >>>>>>>>>> > + NAME=none >>>>>>>>>> > + MVN=/usr/local/spark-latest/build/mvn >>>>>>>>>> > + (( 12 )) >>>>>>>>>> > + case $1 in >>>>>>>>>> > + NAME=spark-latest >>>>>>>>>> > + shift >>>>>>>>>> > + shift >>>>>>>>>> > + (( 10 )) >>>>>>>>>> > + case $1 in >>>>>>>>>> > + MAKE_TGZ=true >>>>>>>>>> > + shift >>>>>>>>>> > + (( 9 )) >>>>>>>>>> > + case $1 in >>>>>>>>>> > + MVN=mvn >>>>>>>>>> > + shift >>>>>>>>>> > + shift >>>>>>>>>> > + (( 7 )) >>>>>>>>>> > + case $1 in >>>>>>>>>> > + break >>>>>>>>>> > + '[' -z >>>>>>>>>> /Library/Java/JavaVirtualMachines/jdk1.8.0_20.jdk/Contents/Home ']' >>>>>>>>>> > + '[' -z >>>>>>>>>> /Library/Java/JavaVirtualMachines/jdk1.8.0_20.jdk/Contents/Home ']' >>>>>>>>>> > ++ command -v git >>>>>>>>>> > + '[' /usr/bin/git ']' >>>>>>>>>> > ++ git rev-parse --short HEAD >>>>>>>>>> > + GITREV=487d409 >>>>>>>>>> > + '[' '!' -z 487d409 ']' >>>>>>>>>> > + GITREVSTRING=' (git revision 487d409)' >>>>>>>>>> > + unset GITREV >>>>>>>>>> > ++ command -v mvn >>>>>>>>>> > + '[' '!' /usr/bin/mvn ']' >>>>>>>>>> > ++ mvn help:evaluate -Dexpression=project.version >>>>>>>>>> -Dhadoop.version=2.7.0 >>>>>>>>>> > -Phadoop-2.7 -Phive -Phive-thriftserver -DskipTests clean >>>>>>>>>> package >>>>>>>>>> > ++ grep -v INFO >>>>>>>>>> > ++ tail -n 1 >>>>>>>>>> > + VERSION='[ERROR] [Help 1] >>>>>>>>>> > >>>>>>>>>> http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException >>>>>>>>>> ' >>>>>>>>>> > >>>>>>>>>> > Same output error with JDK 7 >>>>>>>>>> > >>>>>>>>>> > Appreciate your help. >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>>> >>>>> >>>> >>> >> >
