[
https://issues.apache.org/jira/browse/SPARK-3324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Cheng Lian updated SPARK-3324:
------------------------------
Description:
The YARN module has nonstandard path structure like:
{code}
${SPARK_HOME}
|--yarn
|--alpha (contains yarn api support for 0.23 and 2.0.x)
|--stable (contains yarn api support for 2.2 and later)
| |--pom.xml (spark-yarn)
|--common (Common codes not depending on specific version of Hadoop)
|--pom.xml (yarn-parent)
{code}
When we use maven to compile yarn module, maven will import 'alpha' or 'stable'
module according to profile setting.
And the submodule like 'stable' use the build propertie defined in yarn/pom.xml
to import common codes to sourcePath.
It will cause IntelliJ can't directly recogniz codes in common directory as
sourcePath.
I thought we should change the yarn module to a unified maven jar project,
and choose the version of yarn api via maven profile setting in the pom.xml.
It will resolve the compile error in IntelliJ and make the yarn module more
simple and clear.
was:
The YARN module has nonstandard path structure like:
${SPARK_HOME}
|--yarn
|--alpha (contains yarn api support for 0.23 and 2.0.x)
|--stable (contains yarn api support for 2.2 and later)
| |--pom.xml (spark-yarn)
|--common (Common codes not depending on specific version of Hadoop)
|--pom.xml (yarn-parent)
When we use maven to compile yarn module, maven will import 'alpha' or 'stable'
module according to profile setting.
And the submodule like 'stable' use the build propertie defined in yarn/pom.xml
to import common codes to sourcePath.
It will cause IntelliJ can't directly recogniz codes in common directory as
sourcePath.
I thought we should change the yarn module to a unified maven jar project,
and choose the version of yarn api via maven profile setting in the pom.xml.
It will resolve the compile error in IntelliJ and make the yarn module more
simple and clear.
> YARN module has nonstandard structure which cause compile error In IntelliJ
> ---------------------------------------------------------------------------
>
> Key: SPARK-3324
> URL: https://issues.apache.org/jira/browse/SPARK-3324
> Project: Spark
> Issue Type: Bug
> Components: YARN
> Affects Versions: 1.1.0
> Environment: Mac OS: 10.9.4
> IntelliJ IDEA: 13.1.4
> Scala Plugins: 0.41.2
> Maven: 3.0.5
> Reporter: Yi Tian
> Priority: Minor
> Labels: intellij, maven, yarn
>
> The YARN module has nonstandard path structure like:
> {code}
> ${SPARK_HOME}
> |--yarn
> |--alpha (contains yarn api support for 0.23 and 2.0.x)
> |--stable (contains yarn api support for 2.2 and later)
> | |--pom.xml (spark-yarn)
> |--common (Common codes not depending on specific version of Hadoop)
> |--pom.xml (yarn-parent)
> {code}
> When we use maven to compile yarn module, maven will import 'alpha' or
> 'stable' module according to profile setting.
> And the submodule like 'stable' use the build propertie defined in
> yarn/pom.xml to import common codes to sourcePath.
> It will cause IntelliJ can't directly recogniz codes in common directory as
> sourcePath.
> I thought we should change the yarn module to a unified maven jar project,
> and choose the version of yarn api via maven profile setting in the pom.xml.
> It will resolve the compile error in IntelliJ and make the yarn module more
> simple and clear.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
