[
https://issues.apache.org/jira/browse/FLUME-2819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14972217#comment-14972217
]
Roshan Naik edited comment on FLUME-2819 at 10/24/15 12:52 AM:
---------------------------------------------------------------
Flume generally follows this pattern for its deps (hadoop, hive, hbase)
And no doubt a common issue in other prjs too. For e.g Hive does not ship with
Hadoop deps.
Basically end up relying on the backward binary compat provided by those deps.
Unfortunately due to the complexity it gets tested with only 1 version of each
dependency. Helps a bit that each Hadoop vendor ends up testing it with a
different versions depending on what they ship.
Your suggestion or a variation of it seems like worth looking into. There might
be some tricky things that can happen when including multiple version of
multiple jars. Esp if some of those are fat/bulky jars. Perhaps put all such
deps under a separate lib2 folder.
Many components in Flume will need that change if we go with that...in
addition to startup script needs changes to support for each of those. best
tracked in a separate jira.
Also this is not flume specific situation.
was (Author: roshan_naik):
No doubt a common issue. For e.g Hive does not ship with Hadoop deps.
Basically end up relying on the backward binary compat provided by those deps.
Unfortunately due to the complexity it gets tested with only 1 version of each
dependency. Helps a bit that each Hadoop vendor ends up testing it with a
different versions depending on what they ship.
Your suggestion or a variation of it seems like worth looking into. There might
be some tricky things that can happen when including multiple version of
multiple jars. Esp if some of those are fat/bulky jars. Perhaps put all such
deps under a separate lib2 folder.
Many components in Flume will need that change if we go with that...in
addition to startup script needs changes to support for each of those. best
tracked in a separate jira.
Also this is not flume specific situation.
> Kafka libs are being bundled into Flume distro
> ----------------------------------------------
>
> Key: FLUME-2819
> URL: https://issues.apache.org/jira/browse/FLUME-2819
> Project: Flume
> Issue Type: Bug
> Reporter: Roshan Naik
>
> Kafka dependency libs need to be marked as 'provided' in the pom.xml
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
