[jira] [Commented] (SPARK-4375) Assembly built with Maven is missing most of repl classes
[ https://issues.apache.org/jira/browse/SPARK-4375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14233894#comment-14233894 ] Apache Spark commented on SPARK-4375: - User 'chenghao-intel' has created a pull request for this issue: https://github.com/apache/spark/pull/3595 > Assembly built with Maven is missing most of repl classes > - > > Key: SPARK-4375 > URL: https://issues.apache.org/jira/browse/SPARK-4375 > Project: Spark > Issue Type: Bug >Affects Versions: 1.2.0 >Reporter: Sandy Ryza >Assignee: Sandy Ryza >Priority: Blocker > Fix For: 1.2.0 > > > In particular, the ones in the split scala-2.10/scala-2.11 directories aren't > being added -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-4375) Assembly built with Maven is missing most of repl classes
[ https://issues.apache.org/jira/browse/SPARK-4375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14209433#comment-14209433 ] Apache Spark commented on SPARK-4375: - User 'sryza' has created a pull request for this issue: https://github.com/apache/spark/pull/3239 > Assembly built with Maven is missing most of repl classes > - > > Key: SPARK-4375 > URL: https://issues.apache.org/jira/browse/SPARK-4375 > Project: Spark > Issue Type: Bug >Affects Versions: 1.2.0 >Reporter: Sandy Ryza >Priority: Blocker > > In particular, the ones in the split scala-2.10/scala-2.11 directories aren't > being added -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-4375) Assembly built with Maven is missing most of repl classes
[ https://issues.apache.org/jira/browse/SPARK-4375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14209416#comment-14209416 ] Patrick Wendell commented on SPARK-4375: So my favorite option now is just moving the Kafka example to the kafka directory (it will still be on github, in the source code, etc). Then doing Sandy's trick for the repl project. And for the kafka project make it enabled with a flag only. > Assembly built with Maven is missing most of repl classes > - > > Key: SPARK-4375 > URL: https://issues.apache.org/jira/browse/SPARK-4375 > Project: Spark > Issue Type: Bug >Affects Versions: 1.2.0 >Reporter: Sandy Ryza >Priority: Blocker > > In particular, the ones in the split scala-2.10/scala-2.11 directories aren't > being added -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-4375) Assembly built with Maven is missing most of repl classes
[ https://issues.apache.org/jira/browse/SPARK-4375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14209389#comment-14209389 ] Prashant Sharma commented on SPARK-4375: I was inclined towards not removing that option for one reason, that it shows to our new and seasoned users that there is something like scala version that "NOW" that we have crossbuilding for scala versions, it needs to passed as a command line option. I am saying this realizing that kafka's exclusion is temporary and so on. About producing a helpful message we can probably break the maven build at validation step saying one of the profile is mandatory (If maven allows this, I think I can try !). Thoughts ? > Assembly built with Maven is missing most of repl classes > - > > Key: SPARK-4375 > URL: https://issues.apache.org/jira/browse/SPARK-4375 > Project: Spark > Issue Type: Bug >Affects Versions: 1.2.0 >Reporter: Sandy Ryza >Priority: Blocker > > In particular, the ones in the split scala-2.10/scala-2.11 directories aren't > being added -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-4375) Assembly built with Maven is missing most of repl classes
[ https://issues.apache.org/jira/browse/SPARK-4375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14209388#comment-14209388 ] Sandy Ryza commented on SPARK-4375: --- This all makes sense to me. Will put up a patch. > Assembly built with Maven is missing most of repl classes > - > > Key: SPARK-4375 > URL: https://issues.apache.org/jira/browse/SPARK-4375 > Project: Spark > Issue Type: Bug >Affects Versions: 1.2.0 >Reporter: Sandy Ryza >Priority: Blocker > > In particular, the ones in the split scala-2.10/scala-2.11 directories aren't > being added -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-4375) Assembly built with Maven is missing most of repl classes
[
https://issues.apache.org/jira/browse/SPARK-4375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14209314#comment-14209314
]
Patrick Wendell commented on SPARK-4375:
One thing we could add onto that to make the user have to write less code is
something like this:
{code}
mvn package -Pexamples -Pscala-2.10 -Dexamples-2.10
{code}
The internally we have profiles that are activated by the "examples-2.10"
property and add the relevant modules that are required by the examples build
in 2.10.
> Assembly built with Maven is missing most of repl classes
> -
>
> Key: SPARK-4375
> URL: https://issues.apache.org/jira/browse/SPARK-4375
> Project: Spark
> Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Sandy Ryza
>Priority: Blocker
>
> In particular, the ones in the split scala-2.10/scala-2.11 directories aren't
> being added
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-4375) Assembly built with Maven is missing most of repl classes
[ https://issues.apache.org/jira/browse/SPARK-4375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14209305#comment-14209305 ] Patrick Wendell commented on SPARK-4375: Hey Sandy, What about the following solution: 1. For the repl case, we make the change you are suggesting and simply drop the need for -Pscala-2.10 to be there explicilty. 2. We no longer include the examples module or the external project modules by default in the build. 2. When building the examples, you need to specify, somewhat pedantically, all of the necessary external sub projects and also -Pscala-2.10 or -Pscala-2.11. We can just give people the exact commands to run for 2.10 and 2.11 examples in the maven docs. The main benefits I see is that there is no regression for someone doing a package for Scala 2.10 which is the common case. If someone wants to build the examples, they need to go and do a bit of extra work to look up the new command, but it's mostly straightforward. Of course, all of our packages will still have the examples pre-built. > Assembly built with Maven is missing most of repl classes > - > > Key: SPARK-4375 > URL: https://issues.apache.org/jira/browse/SPARK-4375 > Project: Spark > Issue Type: Bug >Affects Versions: 1.2.0 >Reporter: Sandy Ryza >Priority: Blocker > > In particular, the ones in the split scala-2.10/scala-2.11 directories aren't > being added -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-4375) Assembly built with Maven is missing most of repl classes
[ https://issues.apache.org/jira/browse/SPARK-4375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14209217#comment-14209217 ] Sandy Ryza commented on SPARK-4375: --- The issue here is that the activeByDefault Maven option doesn't work as expected: it turns off all default profiles if any profile is given on the command line. Circumventing this by removing the scala-2.10 profile and setting properties that get overridden when the scala-2.11 profile is turned on is pretty straightforward. Except in a couple cases: * When using Scala 2.11, we don't want to build the external/kafka module * When using Scala 2.11, we don't want the examples module to include the kafka module as a dependency * When using Scala 2.11, we don't want the examples module to include algebird as a dependency As far as I can tell, there aren't Maven doesn't let us do these things in any ways that aren't deeply hacky. For dealing with the first issue, I had talked with Patrick about adding profiles for all of the external modules (including kafka). An issue with this is that the examples depend on all these modules, so we would need to create separate modules or directories for the examples tied to each of these modules. Another thing we could do is add a "scala-2.10-only" profile that turns on building the kafka module and turns on the examples and examples dependencies that only work with scala-2.10. > Assembly built with Maven is missing most of repl classes > - > > Key: SPARK-4375 > URL: https://issues.apache.org/jira/browse/SPARK-4375 > Project: Spark > Issue Type: Bug >Affects Versions: 1.2.0 >Reporter: Sandy Ryza >Priority: Blocker > > In particular, the ones in the split scala-2.10/scala-2.11 directories aren't > being added -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
