[
https://issues.apache.org/jira/browse/HADOOP-8009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14204962#comment-14204962
]
ASF GitHub Bot commented on HADOOP-8009:
----------------------------------------
Github user rmetzger commented on a diff in the pull request:
https://github.com/apache/incubator-flink/pull/172#discussion_r20093842
--- Diff: flink-addons/flink-hbase/pom.xml ---
@@ -116,20 +109,74 @@ under the License.
</exclusions>
</dependency>
</dependencies>
- <!-- <dependency>
- <groupId>org.apache.hbase</groupId>
- <artifactId>hbase-server</artifactId>
- <version>${hbase.version}</version>
- </dependency>
- <dependency>
- <groupId>org.apache.hbase</groupId>
- <artifactId>hbase-client</artifactId>
- <version>${hbase.version}</version>
- </dependency>
- -->
- <!-- hadoop-client is available for yarn and non-yarn, so there is no
need
- to use profiles See ticket
https://issues.apache.org/jira/browse/HADOOP-8009
- for description of hadoop-clients -->
+ <profiles>
+ <profile>
+ <id>hadoop-1</id>
+ <activation>
+ <property>
+ <!-- Please do not remove the 'hadoop1'
comment. See ./tools/generate_specific_pom.sh -->
+ <!--hadoop1 -->
+ <name>!hadoop.profile</name>
+ </property>
+ </activation>
+ <properties>
+
<hbase.version>${hbase.hadoop1.version}</hbase.version>
+ </properties>
+ </profile>
+ <profile>
+ <id>hadoop-2</id>
+ <activation>
+ <property>
+ <!-- Please do not remove the 'hadoop1'
comment. See ./tools/generate_specific_pom.sh -->
+ <!--hadoop2 -->
+ <name>hadoop.profile</name>
+ <value>2</value>
+ </property>
+ </activation>
+ <properties>
+
<hbase.version>${hbase.hadoop2.version}</hbase.version>
+ </properties>
+ <dependencies>
+ <!-- Force hadoop-common dependency -->
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
+ </dependency>
+ </dependencies>
+ </profile>
+ <profile>
+ <id>cdh5.1.3</id>
--- End diff --
What I dislike about the profile is that its very specific about the
version. We basically need to manually maintain the CDH versions and force
users into specific CDH versions.
Would it be possible to add the `<dependencyManagement>` section with the
hadoop-core dependency into the `hadoop2` profile and set the
hadoop.core.version to hadoop-2.2.0 by default?
This way users could actually specify their specific hadoop versions if
they want to build flink against a particular CDH build?
> Create hadoop-client and hadoop-minicluster artifacts for downstream projects
> ------------------------------------------------------------------------------
>
> Key: HADOOP-8009
> URL: https://issues.apache.org/jira/browse/HADOOP-8009
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build
> Affects Versions: 1.0.0, 0.22.0, 0.23.0, 0.23.1, 0.24.0
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Priority: Critical
> Fix For: 1.0.1, 0.23.1
>
> Attachments: HADOOP-8009-branch-0_22.patch,
> HADOOP-8009-branch-1-add.patch, HADOOP-8009-branch-1.patch,
> HADOOP-8009-existing-releases.patch, HADOOP-8009.patch
>
>
> Using Hadoop from projects like Pig/Hive/Sqoop/Flume/Oozie or any in-house
> system that interacts with Hadoop is quite challenging for the following
> reasons:
> * *Different versions of Hadoop produce different artifacts:* Before Hadoop
> 0.23 there was a single artifact hadoop-core, starting with Hadoop 0.23 there
> are several (common, hdfs, mapred*, yarn*)
> * *There are no 'client' artifacts:* Current artifacts include all JARs
> needed to run the services, thus bringing into clients several JARs that are
> not used for job submission/monitoring (servlet, jsp, tomcat, jersey, etc.)
> * *Doing testing on the client side is also quite challenging as more
> artifacts have to be included than the dependencies define:* for example, the
> history-server artifact has to be explicitly included. If using Hadoop 1
> artifacts, jersey-server has to be explicitly included.
> * *3rd party dependencies change in Hadoop from version to version:* This
> makes things complicated for projects that have to deal with multiple
> versions of Hadoop as their exclusions list become a huge mix & match of
> artifacts from different Hadoop versions and it may be break things when a
> particular version of Hadoop requires a dependency that other version of
> Hadoop does not require.
> Because of this it would be quite convenient to have the following
> 'aggregator' artifacts:
> * *org.apache.hadoop:hadoop-client* : it includes all required JARs to use
> Hadoop client APIs (excluding all JARs that are not needed for it)
> * *org.apache.hadoop:hadoop-minicluster* : it includes all required JARs to
> run Hadoop Mini Clusters
> These aggregator artifacts would be created for current branches under
> development (trunk, 0.22, 0.23, 1.0) and for released versions that are still
> in use.
> For branches under development, these artifacts would be generated as part of
> the build.
> For released versions we would have a a special branch used only as vehicle
> for publishing the corresponding 'aggregator' artifacts.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
