[ https://issues.apache.org/jira/browse/FLUME-2819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14972217#comment-14972217 ]
Roshan Naik edited comment on FLUME-2819 at 10/24/15 12:51 AM: --------------------------------------------------------------- No doubt a common issue. For e.g Hive does not ship with Hadoop deps. Basically end up relying on the backward binary compat provided by those deps. Unfortunately due to the complexity it gets tested with only 1 version of each dependency. Helps a bit that each Hadoop vendor ends up testing it with a different versions depending on what they ship. Your suggestion or a variation of it seems like worth looking into. There might be some tricky things that can happen when including multiple version of multiple jars. Esp if some of those are fat/bulky jars. Perhaps put all such deps under a separate lib2 folder. Many components in Flume will need that change if we go with that...in addition to startup script needs changes to support for each of those. best tracked in a separate jira. Also this is not flume specific situation. was (Author: roshan_naik): No doubt a common issue. For e.g Hive does not ship with Hadoop deps. Basically end up relying on the backward binary compat provided by those deps. Unfortunately due to the complexity it gets tested with only 1 version of each dependency. Helps a bit that each Hadoop vendor ends up testing it with a different versions depending on what they ship. Your suggestion or a variation of it seems like worth looking into. There might be some tricky things that can happen when including multiple version of multiple jars. Esp if some of those are fat/bulky jars. Perhaps put all such deps under a separate lib2 folder. Many components in Flume will need that change if we go with that...in addition to startup script needs changes to support for each of those. best tracked in a separate jira. But like you mentioned, this is not flume specific situation. > Kafka libs are being bundled into Flume distro > ---------------------------------------------- > > Key: FLUME-2819 > URL: https://issues.apache.org/jira/browse/FLUME-2819 > Project: Flume > Issue Type: Bug > Reporter: Roshan Naik > > Kafka dependency libs need to be marked as 'provided' in the pom.xml -- This message was sent by Atlassian JIRA (v6.3.4#6332)