Re: Use mvn to build Spark 1.2.0 failed

2015-01-28 Thread Dirceu Semighini Filho 
Before this I was facing the same problem, and fixed it adding the plugin
at the root pom.xml

Maybe this is related to the release, mine is:
Apache Maven 3.2.3 (33f8c3e1027c3ddde99d3cdebad2656a31e8fdf4;
2014-08-11T17:58:10-03:00)
Java version: 1.8.0_20, vendor: Oracle Corporation
OS name: "linux", version: "3.13.0-24-generic", arch: "amd64", family:
"unix"

Or the command that I'm using:
mvn -Dhadoop.version=2.0.0-mr1-cdh4.2.0 -DskipTests -Phive
-Phive-thriftserver clean compile assembly:single

I'm trying to build using the pr/1290
wyphao.2007 have you figured out how to complete the build?



2015-01-28 13:32 GMT-02:00 Sean Owen :

> I don't see how this would relate to the problem in the OP? the
> assemblies build fine already as far as I can tell.
>
> Your new error may be introduced by your change.
>
> On Wed, Jan 28, 2015 at 2:52 PM, Dirceu Semighini Filho
>  wrote:
> > I was facing the same problem, and I fixed it by adding
> >
> > 
> > maven-assembly-plugin
> > 2.4.1
> > 
> >   
> >
>  assembly/src/main/assembly/assembly.xml
> >   
> > 
> > 
> >  in the root pom.xml, following the maven assembly plugin docs
> > <
> http://maven.apache.org/plugins-archives/maven-assembly-plugin-2.4.1/examples/multimodule/module-source-inclusion-simple.html
> >
> >
> > I can make a PR on this if you consider this an issue.
> >
> > Now I'm facing this problem, is that what you have now?
> > [ERROR] Failed to execute goal
> > org.apache.maven.plugins:maven-assembly-plugin:2.4.1:single (default-cli)
> > on project spark-network-common_2.10: Failed to create assembly: Error
> > adding file
> 'org.apache.spark:spark-network-common_2.10:jar:1.3.0-SNAPSHOT'
> > to archive:
> > /home/dirceu/projects/spark/network/common/target/scala-2.10/classes
> isn't
> > a file. -> [Help 1]
> >
> >
> > 2015-01-27 9:23 GMT-02:00 Sean Owen :
> >
> >> You certainly do not need yo build Spark as root. It might clumsily
> >> overcome a permissions problem in your local env but probably causes
> other
> >> problems.
> >> On Jan 27, 2015 11:18 AM, "angel__" 
> >> wrote:
> >>
> >> > I had that problem when I tried to build Spark 1.2. I don't exactly
> know
> >> > what
> >> > is causing it, but I guess it might have something to do with user
> >> > permissions.
> >> >
> >> > I could finally fix this by building Spark as "root" user (now I'm
> >> dealing
> >> > with another problem, but ...that's another story...)
> >> >
> >> >
> >> >
> >> > --
> >> > View this message in context:
> >> >
> >>
> http://apache-spark-developers-list.1001551.n3.nabble.com/Use-mvn-to-build-Spark-1-2-0-failed-tp9876p10285.html
> >> > Sent from the Apache Spark Developers List mailing list archive at
> >> > Nabble.com.
> >> >
> >> > -
> >> > To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
> >> > For additional commands, e-mail: dev-h...@spark.apache.org
> >> >
> >> >
> >>
>

Re: Use mvn to build Spark 1.2.0 failed

2015-01-28 Thread Sean Owen 
I don't see how this would relate to the problem in the OP? the
assemblies build fine already as far as I can tell.

Your new error may be introduced by your change.

On Wed, Jan 28, 2015 at 2:52 PM, Dirceu Semighini Filho
 wrote:
> I was facing the same problem, and I fixed it by adding
>
> 
> maven-assembly-plugin
> 2.4.1
> 
>   
> assembly/src/main/assembly/assembly.xml
>   
> 
> 
>  in the root pom.xml, following the maven assembly plugin docs
> 
>
> I can make a PR on this if you consider this an issue.
>
> Now I'm facing this problem, is that what you have now?
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-assembly-plugin:2.4.1:single (default-cli)
> on project spark-network-common_2.10: Failed to create assembly: Error
> adding file 'org.apache.spark:spark-network-common_2.10:jar:1.3.0-SNAPSHOT'
> to archive:
> /home/dirceu/projects/spark/network/common/target/scala-2.10/classes isn't
> a file. -> [Help 1]
>
>
> 2015-01-27 9:23 GMT-02:00 Sean Owen :
>
>> You certainly do not need yo build Spark as root. It might clumsily
>> overcome a permissions problem in your local env but probably causes other
>> problems.
>> On Jan 27, 2015 11:18 AM, "angel__" 
>> wrote:
>>
>> > I had that problem when I tried to build Spark 1.2. I don't exactly know
>> > what
>> > is causing it, but I guess it might have something to do with user
>> > permissions.
>> >
>> > I could finally fix this by building Spark as "root" user (now I'm
>> dealing
>> > with another problem, but ...that's another story...)
>> >
>> >
>> >
>> > --
>> > View this message in context:
>> >
>> http://apache-spark-developers-list.1001551.n3.nabble.com/Use-mvn-to-build-Spark-1-2-0-failed-tp9876p10285.html
>> > Sent from the Apache Spark Developers List mailing list archive at
>> > Nabble.com.
>> >
>> > -
>> > To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
>> > For additional commands, e-mail: dev-h...@spark.apache.org
>> >
>> >
>>

-
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org

Re: Use mvn to build Spark 1.2.0 failed

2015-01-28 Thread Dirceu Semighini Filho 
I was facing the same problem, and I fixed it by adding


maven-assembly-plugin
2.4.1

  
assembly/src/main/assembly/assembly.xml
  


 in the root pom.xml, following the maven assembly plugin docs


I can make a PR on this if you consider this an issue.

Now I'm facing this problem, is that what you have now?
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-assembly-plugin:2.4.1:single (default-cli)
on project spark-network-common_2.10: Failed to create assembly: Error
adding file 'org.apache.spark:spark-network-common_2.10:jar:1.3.0-SNAPSHOT'
to archive:
/home/dirceu/projects/spark/network/common/target/scala-2.10/classes isn't
a file. -> [Help 1]


2015-01-27 9:23 GMT-02:00 Sean Owen :

> You certainly do not need yo build Spark as root. It might clumsily
> overcome a permissions problem in your local env but probably causes other
> problems.
> On Jan 27, 2015 11:18 AM, "angel__" 
> wrote:
>
> > I had that problem when I tried to build Spark 1.2. I don't exactly know
> > what
> > is causing it, but I guess it might have something to do with user
> > permissions.
> >
> > I could finally fix this by building Spark as "root" user (now I'm
> dealing
> > with another problem, but ...that's another story...)
> >
> >
> >
> > --
> > View this message in context:
> >
> http://apache-spark-developers-list.1001551.n3.nabble.com/Use-mvn-to-build-Spark-1-2-0-failed-tp9876p10285.html
> > Sent from the Apache Spark Developers List mailing list archive at
> > Nabble.com.
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
> > For additional commands, e-mail: dev-h...@spark.apache.org
> >
> >
>

Re: Use mvn to build Spark 1.2.0 failed

2015-01-27 Thread Sean Owen 
You certainly do not need yo build Spark as root. It might clumsily
overcome a permissions problem in your local env but probably causes other
problems.
On Jan 27, 2015 11:18 AM, "angel__"  wrote:

> I had that problem when I tried to build Spark 1.2. I don't exactly know
> what
> is causing it, but I guess it might have something to do with user
> permissions.
>
> I could finally fix this by building Spark as "root" user (now I'm dealing
> with another problem, but ...that's another story...)
>
>
>
> --
> View this message in context:
> http://apache-spark-developers-list.1001551.n3.nabble.com/Use-mvn-to-build-Spark-1-2-0-failed-tp9876p10285.html
> Sent from the Apache Spark Developers List mailing list archive at
> Nabble.com.
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
> For additional commands, e-mail: dev-h...@spark.apache.org
>
>

Re: Use mvn to build Spark 1.2.0 failed

2015-01-27 Thread angel__ 
I had that problem when I tried to build Spark 1.2. I don't exactly know what
is causing it, but I guess it might have something to do with user
permissions.

I could finally fix this by building Spark as "root" user (now I'm dealing
with another problem, but ...that's another story...)



--
View this message in context: 
http://apache-spark-developers-list.1001551.n3.nabble.com/Use-mvn-to-build-Spark-1-2-0-failed-tp9876p10285.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org

Re: Use mvn to build Spark 1.2.0 failed

2014-12-22 Thread Patrick Wendell 
I also couldn't reproduce this issued.

On Mon, Dec 22, 2014 at 2:24 AM, Sean Owen  wrote:
> I just tried the exact same command and do not see any error. Maybe
> you can make sure you're starting from a clean extraction of the
> distro, and check your environment. I'm on OSX, Maven 3.2, Java 8 but
> I don't know that any of those would be relevant.
>
> On Mon, Dec 22, 2014 at 4:10 AM, wyphao.2007  wrote:
>> Hi all, Today download Spark source from 
>> http://spark.apache.org/downloads.html page, and I use
>>
>>
>>  ./make-distribution.sh --tgz -Phadoop-2.2 -Pyarn -DskipTests 
>> -Dhadoop.version=2.2.0 -Phive
>>
>>
>> to build the release, but I encountered an exception as follow:
>>
>>
>> [INFO] --- build-helper-maven-plugin:1.8:add-source (add-scala-sources) @ 
>> spark-parent ---
>> [INFO] Source directory: /home/q/spark/spark-1.2.0/src/main/scala added.
>> [INFO]
>> [INFO] --- maven-remote-resources-plugin:1.5:process (default) @ 
>> spark-parent ---
>> [INFO] 
>> 
>> [INFO] Reactor Summary:
>> [INFO]
>> [INFO] Spark Project Parent POM .. FAILURE [1.015s]
>> [INFO] Spark Project Networking .. SKIPPED
>> [INFO] Spark Project Shuffle Streaming Service ... SKIPPED
>> [INFO] Spark Project Core  SKIPPED
>> [INFO] Spark Project Bagel ... SKIPPED
>> [INFO] Spark Project GraphX .. SKIPPED
>> [INFO] Spark Project Streaming ... SKIPPED
>> [INFO] Spark Project Catalyst  SKIPPED
>> [INFO] Spark Project SQL . SKIPPED
>> [INFO] Spark Project ML Library .. SKIPPED
>> [INFO] Spark Project Tools ... SKIPPED
>> [INFO] Spark Project Hive  SKIPPED
>> [INFO] Spark Project REPL  SKIPPED
>> [INFO] Spark Project YARN Parent POM . SKIPPED
>> [INFO] Spark Project YARN Stable API . SKIPPED
>> [INFO] Spark Project Assembly  SKIPPED
>> [INFO] Spark Project External Twitter  SKIPPED
>> [INFO] Spark Project External Flume Sink . SKIPPED
>> [INFO] Spark Project External Flume .. SKIPPED
>> [INFO] Spark Project External MQTT ... SKIPPED
>> [INFO] Spark Project External ZeroMQ . SKIPPED
>> [INFO] Spark Project External Kafka .. SKIPPED
>> [INFO] Spark Project Examples  SKIPPED
>> [INFO] Spark Project YARN Shuffle Service  SKIPPED
>> [INFO] 
>> 
>> [INFO] BUILD FAILURE
>> [INFO] 
>> 
>> [INFO] Total time: 1.644s
>> [INFO] Finished at: Mon Dec 22 10:56:35 CST 2014
>> [INFO] Final Memory: 21M/481M
>> [INFO] 
>> 
>> [ERROR] Failed to execute goal 
>> org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process (default) 
>> on project spark-parent: Error finding remote resources manifests: 
>> /home/q/spark/spark-1.2.0/target/maven-shared-archive-resources/META-INF/NOTICE
>>  (No such file or directory) -> [Help 1]
>> [ERROR]
>> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
>> switch.
>> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
>> [ERROR]
>> [ERROR] For more information about the errors and possible solutions, please 
>> read the following articles:
>> [ERROR] [Help 1] 
>> http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
>>
>>
>> but the NOTICE file is in the download spark release:
>>
>>
>> [wyp@spark  /home/q/spark/spark-1.2.0]$ ll
>> total 248
>> drwxrwxr-x 3 1000 1000  4096 Dec 10 18:02 assembly
>> drwxrwxr-x 3 1000 1000  4096 Dec 10 18:02 bagel
>> drwxrwxr-x 2 1000 1000  4096 Dec 10 18:02 bin
>> drwxrwxr-x 2 1000 1000  4096 Dec 10 18:02 conf
>> -rw-rw-r-- 1 1000 1000   663 Dec 10 18:02 CONTRIBUTING.md
>> drwxrwxr-x 3 1000 1000  4096 Dec 10 18:02 core
>> drwxrwxr-x 3 1000 1000  4096 Dec 10 18:02 data
>> drwxrwxr-x 4 1000 1000  4096 Dec 10 18:02 dev
>> drwxrwxr-x 3 1000 1000  4096 Dec 10 18:02 docker
>> drwxrwxr-x 7 1000 1000  4096 Dec 10 18:02 docs
>> drwxrwxr-x 4 1000 1000  4096 Dec 10 18:02 ec2
>> drwxrwxr-x 4 1000 1000  4096 Dec 10 18:02 examples
>> drwxrwxr-x 8 1000 1000  4096 Dec 10 18:02 external
>> drwxrwxr-x 5 1000 1000  4096 Dec 10 18:02 extras
>> drwxrwxr-x 4 1000 1000  4096 Dec 10 18:02 graphx
>> -rw-rw-r-- 1 1000 1000 45242 Dec 10 18:02 LICENSE
>> -rwxrwxr-x 1 1000 1000  7941 Dec 10 18:02 make-distribution.sh
>> drwxrwxr-x 3 1000 1000  4096 Dec 10 18:02 mllib
>> drwxrwxr-x 5 1000 1000  4096 D

Re: Use mvn to build Spark 1.2.0 failed

2014-12-22 Thread Sean Owen 
I just tried the exact same command and do not see any error. Maybe
you can make sure you're starting from a clean extraction of the
distro, and check your environment. I'm on OSX, Maven 3.2, Java 8 but
I don't know that any of those would be relevant.

On Mon, Dec 22, 2014 at 4:10 AM, wyphao.2007  wrote:
> Hi all, Today download Spark source from 
> http://spark.apache.org/downloads.html page, and I use
>
>
>  ./make-distribution.sh --tgz -Phadoop-2.2 -Pyarn -DskipTests 
> -Dhadoop.version=2.2.0 -Phive
>
>
> to build the release, but I encountered an exception as follow:
>
>
> [INFO] --- build-helper-maven-plugin:1.8:add-source (add-scala-sources) @ 
> spark-parent ---
> [INFO] Source directory: /home/q/spark/spark-1.2.0/src/main/scala added.
> [INFO]
> [INFO] --- maven-remote-resources-plugin:1.5:process (default) @ spark-parent 
> ---
> [INFO] 
> 
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Spark Project Parent POM .. FAILURE [1.015s]
> [INFO] Spark Project Networking .. SKIPPED
> [INFO] Spark Project Shuffle Streaming Service ... SKIPPED
> [INFO] Spark Project Core  SKIPPED
> [INFO] Spark Project Bagel ... SKIPPED
> [INFO] Spark Project GraphX .. SKIPPED
> [INFO] Spark Project Streaming ... SKIPPED
> [INFO] Spark Project Catalyst  SKIPPED
> [INFO] Spark Project SQL . SKIPPED
> [INFO] Spark Project ML Library .. SKIPPED
> [INFO] Spark Project Tools ... SKIPPED
> [INFO] Spark Project Hive  SKIPPED
> [INFO] Spark Project REPL  SKIPPED
> [INFO] Spark Project YARN Parent POM . SKIPPED
> [INFO] Spark Project YARN Stable API . SKIPPED
> [INFO] Spark Project Assembly  SKIPPED
> [INFO] Spark Project External Twitter  SKIPPED
> [INFO] Spark Project External Flume Sink . SKIPPED
> [INFO] Spark Project External Flume .. SKIPPED
> [INFO] Spark Project External MQTT ... SKIPPED
> [INFO] Spark Project External ZeroMQ . SKIPPED
> [INFO] Spark Project External Kafka .. SKIPPED
> [INFO] Spark Project Examples  SKIPPED
> [INFO] Spark Project YARN Shuffle Service  SKIPPED
> [INFO] 
> 
> [INFO] BUILD FAILURE
> [INFO] 
> 
> [INFO] Total time: 1.644s
> [INFO] Finished at: Mon Dec 22 10:56:35 CST 2014
> [INFO] Final Memory: 21M/481M
> [INFO] 
> 
> [ERROR] Failed to execute goal 
> org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process (default) 
> on project spark-parent: Error finding remote resources manifests: 
> /home/q/spark/spark-1.2.0/target/maven-shared-archive-resources/META-INF/NOTICE
>  (No such file or directory) -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please 
> read the following articles:
> [ERROR] [Help 1] 
> http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
>
>
> but the NOTICE file is in the download spark release:
>
>
> [wyp@spark  /home/q/spark/spark-1.2.0]$ ll
> total 248
> drwxrwxr-x 3 1000 1000  4096 Dec 10 18:02 assembly
> drwxrwxr-x 3 1000 1000  4096 Dec 10 18:02 bagel
> drwxrwxr-x 2 1000 1000  4096 Dec 10 18:02 bin
> drwxrwxr-x 2 1000 1000  4096 Dec 10 18:02 conf
> -rw-rw-r-- 1 1000 1000   663 Dec 10 18:02 CONTRIBUTING.md
> drwxrwxr-x 3 1000 1000  4096 Dec 10 18:02 core
> drwxrwxr-x 3 1000 1000  4096 Dec 10 18:02 data
> drwxrwxr-x 4 1000 1000  4096 Dec 10 18:02 dev
> drwxrwxr-x 3 1000 1000  4096 Dec 10 18:02 docker
> drwxrwxr-x 7 1000 1000  4096 Dec 10 18:02 docs
> drwxrwxr-x 4 1000 1000  4096 Dec 10 18:02 ec2
> drwxrwxr-x 4 1000 1000  4096 Dec 10 18:02 examples
> drwxrwxr-x 8 1000 1000  4096 Dec 10 18:02 external
> drwxrwxr-x 5 1000 1000  4096 Dec 10 18:02 extras
> drwxrwxr-x 4 1000 1000  4096 Dec 10 18:02 graphx
> -rw-rw-r-- 1 1000 1000 45242 Dec 10 18:02 LICENSE
> -rwxrwxr-x 1 1000 1000  7941 Dec 10 18:02 make-distribution.sh
> drwxrwxr-x 3 1000 1000  4096 Dec 10 18:02 mllib
> drwxrwxr-x 5 1000 1000  4096 Dec 10 18:02 network
> -rw-rw-r-- 1 1000 1000 22559 Dec 10 18:02 NOTICE
> -rw-rw-r-- 1 1000 1000 49002 Dec 10 18:02 pom.xml
> drwxrwxr-x 4 1000 1000  4096 Dec 10 18:02 project
> drwxrwxr-x 6 1000 10