On Fri, Mar 20, 2015 at 2:59 PM, Jerry He jerry...@gmail.com wrote:
On Wed, Mar 18, 2015 at 7:38 AM, Jonathan Hsieh j...@cloudera.com wrote:
We are now considered a mature project and as a middle turtle project
we
need to consider the projects that we will inconvenience because
they
is a different story.
Lars
From: Enis Söztutar enis@gmail.com
To: dev@hbase.apache.org dev@hbase.apache.org
Sent: Friday, March 13, 2015 11:56 AM
Subject: Re: [DISCUSS] Dependency compatibility
I think we can solve this generally for Hadoop 2.6.0+. There's no reason
our
. If
not, we can debate whether we have another option, or whether MR/Yarn is a
different story.
-- Lars
From: Enis Söztutar enis@gmail.com
To: dev@hbase.apache.org dev@hbase.apache.org
Sent: Friday, March 13, 2015 11:56 AM
Subject: Re: [DISCUSS] Dependency compatibility
I think
To: dev@hbase.apache.org dev@hbase.apache.org
Sent: Sunday, March 15, 2015 5:14 PM
Subject: Re: [DISCUSS] Dependency compatibility
Sorry for chiming in a bit late.When I wrote up the proposal for our dependency
story I spent a lot of time thinking about this, we also discussed
On Fri, Mar 13, 2015 at 11:59 AM, Andrew Purtell apurt...@apache.org
wrote:
There's no reason our HDFS usage should be exposed in the HBase client
code, and I think the application classpath feature for YARN in that
version can isolate us on the MR side.
I was thinking more the case where
I think we can solve this generally for Hadoop 2.6.0+. There's no reason
our HDFS usage should be exposed in the HBase client code, and I think the
application classpath feature for YARN in that version can isolate us on
the MR side. I am willing to do this work in time for 1.1.
I'm -1 (non-binding) on weakening our compatibility promises. The more we can
isolate our users from the impact of changes upstream the better.
We can't though in general. Making compatibility promises we can't keep
because our upstreams don't (see the dependencies section of Hadoop's
On Fri, Mar 13, 2015 at 11:18 AM, Andrew Purtell apurt...@apache.org
wrote:
I'm -1 (non-binding) on weakening our compatibility promises. The more
we can
isolate our users from the impact of changes upstream the better.
We can't though in general. Making compatibility promises we can't keep
There's no reason our HDFS usage should be exposed in the HBase client code
I did look at this in the past, IIRC, our dependency was we use
hadoop-common code to read our XML configuration files
I would +1 a code duplication to remove the dependency.
I also think it is important for the end
There's no reason our HDFS usage should be exposed in the HBase client
code, and I think the application classpath feature for YARN in that
version can isolate us on the MR side.
I was thinking more the case where we have to bump our version of Guava
because our version and Hadoop's version are
On Thu, Mar 12, 2015 at 1:20 PM, Enis Söztutar enis@gmail.com wrote:
This is good discussion, but I would like to reach a consensus and move on
with HBASE-13149.
My conclusion from the thread is that we cannot be realistically expected
to keep dependency compat between minor versions
I don't think we need a vote unless we cannot reach consensus.
I would be in favor of this change, for the stated reason:
we cannot be realistically expected
to keep dependency compat between minor versions because of lack of
shading in HBase and Hadoop, and our dependencies are themselves
This is good discussion, but I would like to reach a consensus and move on
with HBASE-13149.
My conclusion from the thread is that we cannot be realistically expected
to keep dependency compat between minor versions because of lack of shading
in HBase and Hadoop, and our dependencies are
Always the pragmatist. I accept your proposal with acknowledgement of an
effort to improve the state of things going forward.
On Thursday, March 12, 2015, Enis Söztutar enis@gmail.com wrote:
This is good discussion, but I would like to reach a consensus and move on
with HBASE-13149.
My
When did we retire 0.98 (grin)
Seriously, let's not have Hadoop version specific builds beyond 0.98. Every
other week or so we find and fix breakage in the Hadoop 1 build because
nobody develops with Hadoop 1 or runs it in production. The combinatorics
will be problematic starting with only two
On Wed, Mar 11, 2015 at 4:04 PM, Enis Söztutar e...@apache.org wrote:
1. Hadoop upgraded its version of jackson in 2.5, and HBase MR jobs fail
flat out without corresponding change in HBase (with Hadoop-2.5+). We do
not have any control over Hadoop or similar core dependencies. We cannot be
It's worth noting that if users follow our ref guide (which says to use
hadoop jar), then jobs don't fail. It's only when they attempt to launch
jobs using hbase com.example.MyDriver that things fail.
Additionally, if we stick to telling users that only the hadoop jar
version is supported,
Over at https://issues.apache.org/jira/browse/HBASE-13149, there is some
discussion about changing the jackson version from 1.8 to 1.9 that is worth
bringing here because of the wider audience. The discussion is about
jackson version specific in this issue, but we should also consider future
I'm not a fan of breaking the compat guide lines we set up in our first
minor release. ;(
Can we do something even more generic like shade [1] our dependencies for
some of the common sources of dependency pain like guava, jackson, and
netty? This way we decouple hbase's pain from hadoop's and
On Wed, Mar 11, 2015 at 4:49 PM, Enis Söztutar enis@gmail.com wrote:
It's worth noting that if users follow our ref guide (which says to use
hadoop jar), then jobs don't fail. It's only when they attempt to
launch
jobs using hbase com.example.MyDriver that things fail.
ugh. That should have been updating a single dependency in a breaking way.
On Wed, Mar 11, 2015 at 5:07 PM, Sean Busbey bus...@cloudera.com wrote:
On Wed, Mar 11, 2015 at 4:49 PM, Enis Söztutar enis@gmail.com wrote:
It's worth noting that if users follow our ref guide (which says to
On Wed, Mar 11, 2015 at 5:19 PM, Enis Söztutar enis@gmail.com wrote:
On Wed, Mar 11, 2015 at 3:07 PM, Sean Busbey bus...@cloudera.com wrote:
On Wed, Mar 11, 2015 at 4:49 PM, Enis Söztutar enis@gmail.com
wrote:
It's worth noting that if users follow our ref guide (which says
On Wed, Mar 11, 2015 at 3:07 PM, Sean Busbey bus...@cloudera.com wrote:
On Wed, Mar 11, 2015 at 4:49 PM, Enis Söztutar enis@gmail.com wrote:
It's worth noting that if users follow our ref guide (which says to use
hadoop jar), then jobs don't fail. It's only when they attempt to
ugh, yes please let's not return to the days of hadoop-version-specific
HBase builds.
I'm just spinning up on isolating third-party dependencies over in Hadoop.
I'd be happy to apply whatever works there within HBase. This only really
helps us for HBase 2.y though, right?
Without sidetracking
All of our current ref guide examples use the hadoop jar command:
http://hbase.apache.org/book.html#hbase.mapreduce.classpath
http://hbase.apache.org/book.html#_bundled_hbase_mapreduce_jobs
They only rely on the hbase command to get things that need to be added to
hte hadop classpath.
The advantage of shading/JarJaring our dependencies is we *don't* get stuck
with hbase-A.B-hadoop-Z.Y. Those releases are a hassle for users, and a
hassle for release managers. The disadvantage being runtime bloat in the
form of an excessive number of classes to load.
I think there's a middle
Furthermore, hadoop jar is how you're supposed to launch YARN apps. If
we
say that doing things via the hbase command is acceptable, we're opening
ourselves up to an expansion of what the hbase command has to do. (i.e.
perhaps it should detect if the passed class is a YARN driver and
Hi,
With all those initiatives like ODP, it seems that
we need to build infrastructure for building versions
for major distros of hadoop versions. Currently many
projects do this (spark for example).
So, regardless of what will be in 1.1, should we think about
hbase-hadoop-2.5 or
Can we do something even more generic like shade [1] our dependencies for
some of the common sources of dependency pain like guava, jackson, and
netty? This way we decouple hbase's pain from hadoop's and allow clients
to choose their own versions of libs to use. We'd likely still cause
29 matches
Mail list logo
