Hi Sean
Before compile-time, maven could dynamically add either stable or alpha source
to the yarn/ project.
So there are no incompatible at the compile-time.
Here are an example:
yarn/pom.xml
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>build-helper-maven-plugin</artifactId>
<executions>
<execution>
<id>add-scala-sources</id>
<phase>generate-sources</phase>
<goals>
<goal>add-source</goal>
</goals>
<configuration>
<sources>
<source>common/src/main/scala</source>
<source>${yarn.api}/src/main/scala</source>
</sources>
</configuration>
</execution>
</executions>
</plugin>
On Aug 31, 2014, at 16:19, Sean Owen <[email protected]> wrote:
> This isn't possible since the two versions of YARN are mutually
> incompatible at compile-time. However see my comments about how this
> could be restructured to be a little more standard, and so that
> IntelliJ would parse it out of the box.
>
> Still I imagine it is not worth it if YARN alpha will go away at some
> point and IntelliJ can easily be told where the extra src/ is.
>
> On Sun, Aug 31, 2014 at 3:38 AM, Yi Tian <[email protected]> wrote:
>> Hi everyone!
>>
>> I found the YARN module has nonstandard path structure like:
>>
>> ${SPARK_HOME}
>> |--yarn
>> |--alpha (contains yarn api support for 0.23 and 2.0.x)
>> |--stable (contains yarn api support for 2.2 and later)
>> | |--pom.xml (spark-yarn)
>> |--common (Common codes not depending on specific version of Hadoop)
>> |--pom.xml (yarn-parent)
>>
>> When we use maven to compile yarn module, maven will import 'alpha' or
>> 'stable' module according to profile setting.
>> And the submodule like 'stable' use the build propertie defined in
>> yarn/pom.xml to import common codes to sourcePath.
>> It will cause IntelliJ can't directly recognize sources in common directory
>> as sourcePath.
>>
>> I thought we should change the yarn module to a unified maven jar project,
>> and add specify different version of yarn api via maven profile setting.
>>
>> I created a JIRA ticket: https://issues.apache.org/jira/browse/SPARK-3324
>>
>> Any advice will be appreciated .
>>
>>
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
