Re: Hadoop 2.0 Support for Accumulo 1.4 Branch

[Sean Busbey]

On Fri, Oct 18, 2013 at 12:29 AM, Sean Busbey  wrote:

> On Tue, Oct 15, 2013 at 10:20 AM, Sean Busbey  wrote:
>
>>
>> On Tue, Oct 15, 2013 at 10:16 AM, Sean Busbey wrote:
>>
>>>
>>> On Tue, Oct 15, 2013 at 7:16 AM,  wrote:
>>>
 Just to be clear, we are talking about adding profile support to the
 pom's for Hadoop 2.2.0 for a 1.4.5 and 1.5.1 release, correct? We are not
 talking about changing the default build profile for these branches are we?



>>> for 1.4.5-SNAPSHOT I am only talking about adding support Hadoop 2.2.0.
>>> I am not suggesting we change the default from building against Hadoop
>>> 0.23.203.
>>>
>>>
>>>
>> I mean 0.20.203.0. Ugh, Hadoop versions.
>>
>>
>
> Okay, barring additional suggestions, tomorrow afternoon I'll break things
> down into an umbrella and 3 sub tasks:
>
> 1) addition of hadoop 2 support
>
>  - to include backports of commits
>  - to include making the target hadoop 2 version 2.2.0
>  - to include test changes that flex hadoop 2 features like fail over
>
> 2) ensuring compatibility for 0.20.203
>
> - presuming some subset of the commits in 1) will break it since 0.20
> support was left behind in 1.5
>
> 3) doc / packaging updates
>
> - the issue of binary releases per distro
> - doc patch for what version(s) the release tests are expected to run
> against
>
> Once work is put against those tickets, I'd expect things to go into a
> branch based on the umbrella ticket until such time as the complete work
> can pass the test suite that we'll use at the next release. Then it can get
> rebased onto the 1.4.x dev branch.
>
> --
> Sean
>

Based on recent feedback on ACCUMULO-1792 and ACCUMULO-1795, I want to
resurrect this thread to make sure everyone's concerns are addressed.

For context, here's a link to the start of the last thread:

http://bit.ly/1aPqKuH

>From ACCUMULO-1792, ctubbsii:

> I'd be reluctant to support any Hadoop 2.x support in the 1.4 release
line that breaks compatibility with 0.20. I don't think breaking 0.20
> and then possibly fixing it again as a second step is acceptable (because
that subsequent work may not ever be done, and I don't think
> we should break the compatibility contract that we've established with
1.4.0).

Chris, I believe keeping all of the work in a branch under the umbrella
jira of ACCUMULO-1790 will ensure that we don't end up with a 1.4 release
that doesn't have proper support for 0.20.203.

Is there something beyond making sure the branch passes a full set of
release tests on 0.20.203 that you'd like to see? In the event that the
branch only ever contains the work for adding Hadoop 2, it's a simple
matter to abandon without rolling into the 1.4 development line.

>From ACCUMULO-1795, bills (and +1ed by elserj and ctubbsii):

> I'm very uncomfortable with risking breaking continuity in such an old
release, and I don't think managing two lines of 1.4 releases is
> worth the effort. Though we have no official EOL policy, 1.3 was
practically dead in the water once 1.4 was around, and I hope we start
> encouraging more adoption of 1.5 (and soon 1.6) versus continually
propping up 1.4.

I'd love to get people to move off of 1.4. However, I think adding Hadoop 2
support to 1.4 encourages this more than leaving it out.

Accumulo 1.5.x places a higher burden on HDFS than 1.4 did, and I'm not
surprised people find relying on 0.20 for the 1.5 WAL intimidating.
Upgrading both HDFS and Accumulo across major versions at once is asking
them to take on a bunch of risk. By adding in Hadoop 2 support to 1.4 we
allow them to break the risk up into steps: they can upgrade HDFS versions
first, get comfortable, then upgrade Accumulo to 1.5.

I think the existing tickets under the umbrella of ACCUMULO-1790 should
ensure that we end up with a single 1.4 line that can work with either the
existing 0.20.203.0 claimed in releases or against 2.2.0.

Bill (or Josh or Chris), is there stronger language you'd like to see
around docs / packaging (area #3 in the original plan and currently
ACCUMULO-1796)? Maybe expressly only doing a binary convenience package for
0.20.203.0? Are you looking for something beyond a full release suite to
ensure 1.4 is still maintaining compatibility on Hadoop 0.20.203?


-Sean

Review Request 15457: ACCUMULO-1884 NativeMap compilation on Mac OS X, 1.6.x

[Bill Havanki]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15457/
---

Review request for accumulo.


Bugs: ACCUMULO-1884
https://issues.apache.org/jira/browse/ACCUMULO-1884


Repository: accumulo


Description
---

Updates to the NativeMap Makefile for successful build under Mac OS X. Repeats 
work done in ACCUMULO-1819.


Diffs
-

  server/native/src/main/resources/Makefile 
1c1f872f52ba0e3fd3827574256590233c73616d 

Diff: https://reviews.apache.org/r/15457/diff/


Testing
---

Ran build on Mac OS X 10.8.5 (Mountain Lion). Without changes, the build fails 
to find the JNI header et al.


Thanks,

Bill Havanki

Blog for Accumulo?

[Bill Havanki]

Apache hosts blogs for many of its projects. For example:

http://blogs.apache.org/hbase/

Any interest in establishing one for Accumulo?

Bill

-- 
| - - -
| Bill Havanki
| Solutions Architect, Cloudera Government Solutions
| - - -

Re: Blog for Accumulo?

[David Medinets]

Who would contribute? How many Accumulo-focused blogs are there currently?
Would the 'official' blog be one author or multiple?


On Tue, Nov 12, 2013 at 11:43 AM, Bill Havanki wrote:

> Apache hosts blogs for many of its projects. For example:
>
> http://blogs.apache.org/hbase/
>
> Any interest in establishing one for Accumulo?
>
> Bill
>
> --
> | - - -
> | Bill Havanki
> | Solutions Architect, Cloudera Government Solutions
> | - - -
>

Re: Blog for Accumulo?

[Bill Havanki]

Ah, found this: http://www.apache.org/dev/project-blogs

My preferences:
- anyone can contribute
- the official blog would have multiple authors, whose content is posted on
their behalf by committers
- blog post content is discussed / vetted by the PMC before release (to
some degree)

Googling for "accumulo blog" returns a bunch of blogs out there, but the
"official" one need not compete with them.

Bill



On Tue, Nov 12, 2013 at 12:04 PM, David Medinets
wrote:

> Who would contribute? How many Accumulo-focused blogs are there currently?
> Would the 'official' blog be one author or multiple?
>
>
> On Tue, Nov 12, 2013 at 11:43 AM, Bill Havanki  >wrote:
>
> > Apache hosts blogs for many of its projects. For example:
> >
> > http://blogs.apache.org/hbase/
> >
> > Any interest in establishing one for Accumulo?
> >
> > Bill
> >
> > --
> > | - - -
> > | Bill Havanki
> > | Solutions Architect, Cloudera Government Solutions
> > | - - -
> >
>



-- 
| - - -
| Bill Havanki
| Solutions Architect, Cloudera Government Solutions
| - - -

Re: Review Request 15457: ACCUMULO-1884 NativeMap compilation on Mac OS X, 1.6.x

[Eric Newton]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15457/#review28742
---

Ship it!


Ship It!

- Eric Newton


On Nov. 12, 2013, 4:21 p.m., Bill Havanki wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/15457/
> ---
> 
> (Updated Nov. 12, 2013, 4:21 p.m.)
> 
> 
> Review request for accumulo.
> 
> 
> Bugs: ACCUMULO-1884
> https://issues.apache.org/jira/browse/ACCUMULO-1884
> 
> 
> Repository: accumulo
> 
> 
> Description
> ---
> 
> Updates to the NativeMap Makefile for successful build under Mac OS X. 
> Repeats work done in ACCUMULO-1819.
> 
> 
> Diffs
> -
> 
>   server/native/src/main/resources/Makefile 
> 1c1f872f52ba0e3fd3827574256590233c73616d 
> 
> Diff: https://reviews.apache.org/r/15457/diff/
> 
> 
> Testing
> ---
> 
> Ran build on Mac OS X 10.8.5 (Mountain Lion). Without changes, the build 
> fails to find the JNI header et al.
> 
> 
> Thanks,
> 
> Bill Havanki
> 
>

Re: Blog for Accumulo?

[Billie Rinaldi]

I think that we didn't create a blog initially because we only wanted to do
it if we had someone (ideally more than one someone) committed to blogging
regularly.  I'm open to discussing it again.  Do you think there is an
advantage of having an official blog, rather than linking to people's
individual blog posts from the Accumulo web site (
http://accumulo.apache.org/papers.html)?


On Tue, Nov 12, 2013 at 8:43 AM, Bill Havanki wrote:

> Apache hosts blogs for many of its projects. For example:
>
> http://blogs.apache.org/hbase/
>
> Any interest in establishing one for Accumulo?
>
> Bill
>
> --
> | - - -
> | Bill Havanki
> | Solutions Architect, Cloudera Government Solutions
> | - - -
>

Re: Blog for Accumulo?

[David Medinets]

1. The link (paper.html) seems like it could change to links.html to better
fit the content. Although that leads to broken link syndrome.
2. Under the Blog posts, consider giving MapR its own sub-list. Two links
on one line is hard for me to see.
3. Please change the Accumulo on Ubuntu
VirtualBoxlink to refer my Vagrant
projects. They are more up-to-date and way easier
to use.
  https://github.com/medined/Accumulo_1_4_4_By_Vagrant
  https://github.com/medined/Accumulo_1_5_0_By_Vagrant
  https://github.com/medined/Accumulo_Snapshot_By_Vagrant

Sorry to thread-jack...


On Tue, Nov 12, 2013 at 12:21 PM, Billie Rinaldi
wrote:

> I think that we didn't create a blog initially because we only wanted to do
> it if we had someone (ideally more than one someone) committed to blogging
> regularly.  I'm open to discussing it again.  Do you think there is an
> advantage of having an official blog, rather than linking to people's
> individual blog posts from the Accumulo web site (
> http://accumulo.apache.org/papers.html)?
>
>
> On Tue, Nov 12, 2013 at 8:43 AM, Bill Havanki  >wrote:
>
> > Apache hosts blogs for many of its projects. For example:
> >
> > http://blogs.apache.org/hbase/
> >
> > Any interest in establishing one for Accumulo?
> >
> > Bill
> >
> > --
> > | - - -
> > | Bill Havanki
> > | Solutions Architect, Cloudera Government Solutions
> > | - - -
> >
>

Re: Blog for Accumulo?

[Bill Havanki]

Here are some thoughts:

* Links to individual blog posts could be cross-posted to the official
blog, and I imagine that's a lighter-weight operation than updating the
papers.html page. The official blog could become the one-stop shop for
Accumulo posts, whether local or cross-posted.
* The official blog provides a space for posts that don't make sense on
individual blogs, such as announcing releases and soliciting feedback.
* An official blog would lower the barrier to entry for contributors; they
would not have to stand up their own technical blog if they don't want to.


On Tue, Nov 12, 2013 at 12:21 PM, Billie Rinaldi
wrote:

> I think that we didn't create a blog initially because we only wanted to do
> it if we had someone (ideally more than one someone) committed to blogging
> regularly.  I'm open to discussing it again.  Do you think there is an
> advantage of having an official blog, rather than linking to people's
> individual blog posts from the Accumulo web site (
> http://accumulo.apache.org/papers.html)?
>
>
> On Tue, Nov 12, 2013 at 8:43 AM, Bill Havanki  >wrote:
>
> > Apache hosts blogs for many of its projects. For example:
> >
> > http://blogs.apache.org/hbase/
> >
> > Any interest in establishing one for Accumulo?
> >
> > Bill
> >
> > --
> > | - - -
> > | Bill Havanki
> > | Solutions Architect, Cloudera Government Solutions
> > | - - -
> >
>



-- 
| - - -
| Bill Havanki
| Solutions Architect, Cloudera Government Solutions
| - - -

Re: Blog for Accumulo?

[Ravi Mutyala]

I like to see an official blog too. It could have cross posting from other
blogs or links to blog posts that we want all users to see.

On a side note, both the MapR on links page are broken. One gives a 404 and
other one takes us to website and not to the blog post.


On Tue, Nov 12, 2013 at 11:50 AM, David Medinets
wrote:

> 1. The link (paper.html) seems like it could change to links.html to better
> fit the content. Although that leads to broken link syndrome.
> 2. Under the Blog posts, consider giving MapR its own sub-list. Two links
> on one line is hard for me to see.
> 3. Please change the Accumulo on Ubuntu
> VirtualBoxlink to refer my Vagrant
> projects. They are more up-to-date and way easier
> to use.
>   https://github.com/medined/Accumulo_1_4_4_By_Vagrant
>   https://github.com/medined/Accumulo_1_5_0_By_Vagrant
>   https://github.com/medined/Accumulo_Snapshot_By_Vagrant
>
> Sorry to thread-jack...
>
>
> On Tue, Nov 12, 2013 at 12:21 PM, Billie Rinaldi
> wrote:
>
> > I think that we didn't create a blog initially because we only wanted to
> do
> > it if we had someone (ideally more than one someone) committed to
> blogging
> > regularly.  I'm open to discussing it again.  Do you think there is an
> > advantage of having an official blog, rather than linking to people's
> > individual blog posts from the Accumulo web site (
> > http://accumulo.apache.org/papers.html)?
> >
> >
> > On Tue, Nov 12, 2013 at 8:43 AM, Bill Havanki  > >wrote:
> >
> > > Apache hosts blogs for many of its projects. For example:
> > >
> > > http://blogs.apache.org/hbase/
> > >
> > > Any interest in establishing one for Accumulo?
> > >
> > > Bill
> > >
> > > --
> > > | - - -
> > > | Bill Havanki
> > > | Solutions Architect, Cloudera Government Solutions
> > > | - - -
> > >
> >
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: Review Request 8559: Add Accumulo support to Sqoop

[Jarek Cecho]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8559/#review28593
---


Hi Philip,
I was playing with the patch a bit and I have to say, very good job! Unit tests 
in all profiles are passing, which is great. I'm still trying to test the patch 
on a real Hadoop/Accumulo cluster. I'll keep you posted about my progress 
there. Couple of notes that I've noticed:

1) You've put together great docs that are very easy to follow, thank you! They 
are however not being included in user guide as they are referenced anywhere. 
Would you mind ensuring that the docs will get propagated into the User guide 
as well? You can use command ant docs to generate the User guide.

2) For Hive and HBase import we do have special bash code in the loading 
scripts that are ensuring that all the required dependencies are on classpath. 
Would you mind providing similar solution for Accumulo, so that users don't 
have to hack the classpath together themselves?


src/java/org/apache/sqoop/tool/BaseSqoopTool.java


Seems like unused import.


Jarcec

- Jarek Cecho


On Nov. 11, 2013, 3:31 p.m., Philip Grim wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/8559/
> ---
> 
> (Updated Nov. 11, 2013, 3:31 p.m.)
> 
> 
> Review request for accumulo, Sqoop and Jarek Cecho.
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> Adds the ability to import to an Accumulo table in much the same manner as 
> the current HBase import capability.  Reported in JIRA as ACCUMULO-141 and 
> SQOOP-767.
> 
> 
> Diffs
> -
> 
>   ivy.xml c5130ae 
>   src/docs/user/accumulo-args.txt PRE-CREATION 
>   src/docs/user/accumulo.txt PRE-CREATION 
>   src/java/org/apache/sqoop/SqoopOptions.java 5c7a56a 
>   src/java/org/apache/sqoop/accumulo/AccumuloConstants.java PRE-CREATION 
>   src/java/org/apache/sqoop/accumulo/AccumuloMutationProcessor.java 
> PRE-CREATION 
>   src/java/org/apache/sqoop/accumulo/AccumuloUtil.java PRE-CREATION 
>   src/java/org/apache/sqoop/accumulo/MutationTransformer.java PRE-CREATION 
>   src/java/org/apache/sqoop/accumulo/ToStringMutationTransformer.java 
> PRE-CREATION 
>   src/java/org/apache/sqoop/manager/SqlManager.java 1ffa40f 
>   src/java/org/apache/sqoop/mapreduce/AccumuloImportJob.java PRE-CREATION 
>   src/java/org/apache/sqoop/mapreduce/AccumuloImportMapper.java PRE-CREATION 
>   src/java/org/apache/sqoop/tool/BaseSqoopTool.java 018d11f 
>   src/java/org/apache/sqoop/tool/ImportTool.java fbbde1d 
>   src/test/org/apache/sqoop/accumulo/AccumuloTestCase.java PRE-CREATION 
>   src/test/org/apache/sqoop/accumulo/TestAccumuloImport.java PRE-CREATION 
>   src/test/org/apache/sqoop/accumulo/TestAccumuloQueryImport.java 
> PRE-CREATION 
>   src/test/org/apache/sqoop/accumulo/TestAccumuloUtil.java PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/8559/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Philip Grim
> 
>

Re: Hadoop 2.0 Support for Accumulo 1.4 Branch

[Josh Elser]

Based on recent feedback on ACCUMULO-1792 and ACCUMULO-1795, I want to
resurrect this thread to make sure everyone's concerns are addressed.

For context, here's a link to the start of the last thread:

http://bit.ly/1aPqKuH

 From ACCUMULO-1792, ctubbsii:


I'd be reluctant to support any Hadoop 2.x support in the 1.4 release

line that breaks compatibility with 0.20. I don't think breaking 0.20

and then possibly fixing it again as a second step is acceptable (because

that subsequent work may not ever be done, and I don't think

we should break the compatibility contract that we've established with

1.4.0).

Chris, I believe keeping all of the work in a branch under the umbrella
jira of ACCUMULO-1790 will ensure that we don't end up with a 1.4 release
that doesn't have proper support for 0.20.203.

Is there something beyond making sure the branch passes a full set of
release tests on 0.20.203 that you'd like to see? In the event that the
branch only ever contains the work for adding Hadoop 2, it's a simple
matter to abandon without rolling into the 1.4 development line.

 From ACCUMULO-1795, bills (and +1ed by elserj and ctubbsii):


I'm very uncomfortable with risking breaking continuity in such an old

release, and I don't think managing two lines of 1.4 releases is

worth the effort. Though we have no official EOL policy, 1.3 was

practically dead in the water once 1.4 was around, and I hope we start

encouraging more adoption of 1.5 (and soon 1.6) versus continually

propping up 1.4.

I'd love to get people to move off of 1.4. However, I think adding Hadoop 2
support to 1.4 encourages this more than leaving it out.


I'm not sure I agree that adding Hadoop2 support to 1.4 encourages 
people to upgrade Accumulo. My gut reaction would be that it allows 
people to completely ignore Accumulo updates (ignoring moving to 1.4.5 
which would allow them to do hadoop2 with your proposed changes)



Accumulo 1.5.x places a higher burden on HDFS than 1.4 did, and I'm not
surprised people find relying on 0.20 for the 1.5 WAL intimidating.
Upgrading both HDFS and Accumulo across major versions at once is asking
them to take on a bunch of risk. By adding in Hadoop 2 support to 1.4 we
allow them to break the risk up into steps: they can upgrade HDFS versions
first, get comfortable, then upgrade Accumulo to 1.5.


Personally, maintaining 0.20 compatibility is not a big concern on my 
radar. If you're still running an 0.20 release, I'd *really* hope that 
you have an upgrade path to 1.2.x (if not 2.2.x) scheduled.


I think claiming that 1.5 has a higher burden on 1.4 is a bit of a 
fallacy. There were many problems and pains regarding WALs in <=1.4 that 
are very difficult to work with in a large environment (try finding WALs 
in server failure cases). I think the increased I/O on HDFS is a much 
smaller cost than the completely different I/O path that the old loggers 
have.


I also think upgrading Accumulo is much less scary than upgrading HDFS, 
but that's just me.


To me, it seems like the argument may be coming down to whether or not 
we break 0.20 hadoop compatibility on a bug-fix release and how 
concerned we are about letting users lag behind the upstream development.



I think the existing tickets under the umbrella of ACCUMULO-1790 should
ensure that we end up with a single 1.4 line that can work with either the
existing 0.20.203.0 claimed in releases or against 2.2.0.

Bill (or Josh or Chris), is there stronger language you'd like to see
around docs / packaging (area #3 in the original plan and currently
ACCUMULO-1796)? Maybe expressly only doing a binary convenience package for
0.20.203.0? Are you looking for something beyond a full release suite to
ensure 1.4 is still maintaining compatibility on Hadoop 0.20.203?



Again, my biggest concern here is not following our own guidelines of 
breaking changes across minor releases, but I'd hope 0.20 users have an 
upgrade path outlined for themselves.

Re: Review Request 15457: ACCUMULO-1884 NativeMap compilation on Mac OS X, 1.6.x

[Josh Elser]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15457/#review28748
---


This isn't a complete fix. On Mavericks, `mvn verify` does not succeed -- see 
ACCUMULO-1852. 

- Josh Elser


On Nov. 12, 2013, 4:21 p.m., Bill Havanki wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/15457/
> ---
> 
> (Updated Nov. 12, 2013, 4:21 p.m.)
> 
> 
> Review request for accumulo.
> 
> 
> Bugs: ACCUMULO-1884
> https://issues.apache.org/jira/browse/ACCUMULO-1884
> 
> 
> Repository: accumulo
> 
> 
> Description
> ---
> 
> Updates to the NativeMap Makefile for successful build under Mac OS X. 
> Repeats work done in ACCUMULO-1819.
> 
> 
> Diffs
> -
> 
>   server/native/src/main/resources/Makefile 
> 1c1f872f52ba0e3fd3827574256590233c73616d 
> 
> Diff: https://reviews.apache.org/r/15457/diff/
> 
> 
> Testing
> ---
> 
> Ran build on Mac OS X 10.8.5 (Mountain Lion). Without changes, the build 
> fails to find the JNI header et al.
> 
> 
> Thanks,
> 
> Bill Havanki
> 
>

Re: Hadoop 2.0 Support for Accumulo 1.4 Branch

[William Slacum]

A user of 1.4.a should be able to move to 1.4.b without any "major"
infrastructure changes, such as swapping out HDFS or installing extra
add-ons.

I don't find much merit in debating local WAL vs HDFS WAL cost/benefit
since the only quantifiable evidence we have supported the move.

I should note, Sean, that if you see merit in the work, you don't need
community approval for forking and sharing. However, I do not think it is
in the community's best interest to continue to upgrade 1.4.



On Tue, Nov 12, 2013 at 2:12 PM, Josh Elser  wrote:

>
>> Based on recent feedback on ACCUMULO-1792 and ACCUMULO-1795, I want to
>> resurrect this thread to make sure everyone's concerns are addressed.
>>
>> For context, here's a link to the start of the last thread:
>>
>> http://bit.ly/1aPqKuH
>>
>>  From ACCUMULO-1792, ctubbsii:
>>
>>  I'd be reluctant to support any Hadoop 2.x support in the 1.4 release
>>>
>> line that breaks compatibility with 0.20. I don't think breaking 0.20
>>
>>> and then possibly fixing it again as a second step is acceptable (because
>>>
>> that subsequent work may not ever be done, and I don't think
>>
>>> we should break the compatibility contract that we've established with
>>>
>> 1.4.0).
>>
>> Chris, I believe keeping all of the work in a branch under the umbrella
>> jira of ACCUMULO-1790 will ensure that we don't end up with a 1.4 release
>> that doesn't have proper support for 0.20.203.
>>
>> Is there something beyond making sure the branch passes a full set of
>> release tests on 0.20.203 that you'd like to see? In the event that the
>> branch only ever contains the work for adding Hadoop 2, it's a simple
>> matter to abandon without rolling into the 1.4 development line.
>>
>>  From ACCUMULO-1795, bills (and +1ed by elserj and ctubbsii):
>>
>>  I'm very uncomfortable with risking breaking continuity in such an old
>>>
>> release, and I don't think managing two lines of 1.4 releases is
>>
>>> worth the effort. Though we have no official EOL policy, 1.3 was
>>>
>> practically dead in the water once 1.4 was around, and I hope we start
>>
>>> encouraging more adoption of 1.5 (and soon 1.6) versus continually
>>>
>> propping up 1.4.
>>
>> I'd love to get people to move off of 1.4. However, I think adding Hadoop
>> 2
>> support to 1.4 encourages this more than leaving it out.
>>
>
> I'm not sure I agree that adding Hadoop2 support to 1.4 encourages people
> to upgrade Accumulo. My gut reaction would be that it allows people to
> completely ignore Accumulo updates (ignoring moving to 1.4.5 which would
> allow them to do hadoop2 with your proposed changes)
>
>
>  Accumulo 1.5.x places a higher burden on HDFS than 1.4 did, and I'm not
>> surprised people find relying on 0.20 for the 1.5 WAL intimidating.
>> Upgrading both HDFS and Accumulo across major versions at once is asking
>> them to take on a bunch of risk. By adding in Hadoop 2 support to 1.4 we
>> allow them to break the risk up into steps: they can upgrade HDFS versions
>> first, get comfortable, then upgrade Accumulo to 1.5.
>>
>
> Personally, maintaining 0.20 compatibility is not a big concern on my
> radar. If you're still running an 0.20 release, I'd *really* hope that you
> have an upgrade path to 1.2.x (if not 2.2.x) scheduled.
>
> I think claiming that 1.5 has a higher burden on 1.4 is a bit of a
> fallacy. There were many problems and pains regarding WALs in <=1.4 that
> are very difficult to work with in a large environment (try finding WALs in
> server failure cases). I think the increased I/O on HDFS is a much smaller
> cost than the completely different I/O path that the old loggers have.
>
> I also think upgrading Accumulo is much less scary than upgrading HDFS,
> but that's just me.
>
> To me, it seems like the argument may be coming down to whether or not we
> break 0.20 hadoop compatibility on a bug-fix release and how concerned we
> are about letting users lag behind the upstream development.
>
>
>  I think the existing tickets under the umbrella of ACCUMULO-1790 should
>> ensure that we end up with a single 1.4 line that can work with either the
>> existing 0.20.203.0 claimed in releases or against 2.2.0.
>>
>> Bill (or Josh or Chris), is there stronger language you'd like to see
>> around docs / packaging (area #3 in the original plan and currently
>> ACCUMULO-1796)? Maybe expressly only doing a binary convenience package
>> for
>> 0.20.203.0? Are you looking for something beyond a full release suite to
>> ensure 1.4 is still maintaining compatibility on Hadoop 0.20.203?
>>
>>
> Again, my biggest concern here is not following our own guidelines of
> breaking changes across minor releases, but I'd hope 0.20 users have an
> upgrade path outlined for themselves.
>

ACCUMULO-1870 - Committer Review Request

[Ryan Fishel]

Hello!

JIRA ticket ACCUMULO-1870 was opened late last week. The fix was fairly
trivial and, as of yesterday, a patch has been created. Would any
committers have some time to take a look and approve it?

Thanks!
Ryan Fishel

Re: Hadoop 2.0 Support for Accumulo 1.4 Branch

[Sean Busbey]

On Tue, Nov 12, 2013 at 1:12 PM, Josh Elser  wrote:

>
>>
> To me, it seems like the argument may be coming down to whether or not we
> break 0.20 hadoop compatibility on a bug-fix release and how concerned we
> are about letting users lag behind the upstream development.
>
>
>  I think the existing tickets under the umbrella of ACCUMULO-1790 should
>> ensure that we end up with a single 1.4 line that can work with either the
>> existing 0.20.203.0 claimed in releases or against 2.2.0.
>>
>> Bill (or Josh or Chris), is there stronger language you'd like to see
>> around docs / packaging (area #3 in the original plan and currently
>> ACCUMULO-1796)? Maybe expressly only doing a binary convenience package
>> for
>> 0.20.203.0? Are you looking for something beyond a full release suite to
>> ensure 1.4 is still maintaining compatibility on Hadoop 0.20.203?
>>
>>
> Again, my biggest concern here is not following our own guidelines of
> breaking changes across minor releases, but I'd hope 0.20 users have an
> upgrade path outlined for themselves.
>


The plan outlined in the original thread, and in the subtasks under
ACCUMULO-1790, is expressly aimed at not breaking 0.20 compatibility in the
1.4 bugfix line. If there's anything we can do besides running through the
release test suite on a 0.20 cluster to help ensure that, I am interested
in adding it to the existing plan.


-- 
Sean

Re: Hadoop 2.0 Support for Accumulo 1.4 Branch

[Sean Busbey]

On Tue, Nov 12, 2013 at 1:28 PM, William Slacum <
wilhelm.von.cl...@accumulo.net> wrote:

> A user of 1.4.a should be able to move to 1.4.b without any "major"
> infrastructure changes, such as swapping out HDFS or installing extra
> add-ons.
>
>

Right, exactly. Hopefully no part of the original plan contradicts this. Is
there something that appears to?


-- 
Sean

Re: Hadoop 2.0 Support for Accumulo 1.4 Branch

[Josh Elser]

On 11/12/13, 12:24 PM, Sean Busbey wrote:

On Tue, Nov 12, 2013 at 1:12 PM, Josh Elser  wrote:






To me, it seems like the argument may be coming down to whether or not we
break 0.20 hadoop compatibility on a bug-fix release and how concerned we
are about letting users lag behind the upstream development.


  I think the existing tickets under the umbrella of ACCUMULO-1790 should

ensure that we end up with a single 1.4 line that can work with either the
existing 0.20.203.0 claimed in releases or against 2.2.0.

Bill (or Josh or Chris), is there stronger language you'd like to see
around docs / packaging (area #3 in the original plan and currently
ACCUMULO-1796)? Maybe expressly only doing a binary convenience package
for
0.20.203.0? Are you looking for something beyond a full release suite to
ensure 1.4 is still maintaining compatibility on Hadoop 0.20.203?



Again, my biggest concern here is not following our own guidelines of
breaking changes across minor releases, but I'd hope 0.20 users have an
upgrade path outlined for themselves.




The plan outlined in the original thread, and in the subtasks under
ACCUMULO-1790, is expressly aimed at not breaking 0.20 compatibility in the
1.4 bugfix line. If there's anything we can do besides running through the
release test suite on a 0.20 cluster to help ensure that, I am interested
in adding it to the existing plan.




What about the other half: "encouraging" users to lag (soon to be) two 
major releases behind?

Re: Hadoop 2.0 Support for Accumulo 1.4 Branch

[William Slacum]

The language of ACCUMULO-1795 indicated that an acceptable state was
something that wasn't binary compatible. That's my #1 thing to avoid.

> Maybe expressly only doing a binary convenience package for
> 0.20.203.0?

If we need an extra package, doesn't that mean a user can't just upgrade
Accumulo?

As a side note, 0.20.203.0 is 1.4,

On Tue, Nov 12, 2013 at 3:28 PM, Sean Busbey wrote:

> On Tue, Nov 12, 2013 at 1:28 PM, William Slacum <
> wilhelm.von.cl...@accumulo.net> wrote:
>
> > A user of 1.4.a should be able to move to 1.4.b without any "major"
> > infrastructure changes, such as swapping out HDFS or installing extra
> > add-ons.
> >
> >
>
> Right, exactly. Hopefully no part of the original plan contradicts this. Is
> there something that appears to?
>
>
> --
> Sean
>

Re: Blog for Accumulo?

[Josh Elser]

I'm all for having a more "definitive" location which we can use as a 
distribution point for accurate Accumulo information.


We might be able to leverage the ASF's planet instance 
(http://planet.apache.org/) as a aggregation service and link some 
subset of posts to planet on a.a.o? I'm thinking like make a sub-feed of 
posts to planet.apache.org/committers which has an accumulo tag (I'm not 
sure if that's directly possible though).


On 11/12/13, 8:43 AM, Bill Havanki wrote:

Apache hosts blogs for many of its projects. For example:

http://blogs.apache.org/hbase/

Any interest in establishing one for Accumulo?

Bill

Re: Hadoop 2.0 Support for Accumulo 1.4 Branch

[William Slacum]

herp, ignore that "As a side note" line. I was crafting something else and
came back later. Forgot to clean it up.


On Tue, Nov 12, 2013 at 4:14 PM, William Slacum <
wilhelm.von.cl...@accumulo.net> wrote:

> The language of ACCUMULO-1795 indicated that an acceptable state was
> something that wasn't binary compatible. That's my #1 thing to avoid.
>
> > Maybe expressly only doing a binary convenience package for
> > 0.20.203.0?
>
> If we need an extra package, doesn't that mean a user can't just upgrade
> Accumulo?
>
> As a side note, 0.20.203.0 is 1.4,
>
>
> On Tue, Nov 12, 2013 at 3:28 PM, Sean Busbey 
> wrote:
>
>> On Tue, Nov 12, 2013 at 1:28 PM, William Slacum <
>> wilhelm.von.cl...@accumulo.net> wrote:
>>
>> > A user of 1.4.a should be able to move to 1.4.b without any "major"
>> > infrastructure changes, such as swapping out HDFS or installing extra
>> > add-ons.
>> >
>> >
>>
>> Right, exactly. Hopefully no part of the original plan contradicts this.
>> Is
>> there something that appears to?
>>
>>
>> --
>> Sean
>>
>
>

Re: Hadoop 2.0 Support for Accumulo 1.4 Branch

[Sean Busbey]

On Tue, Nov 12, 2013 at 2:48 PM, Josh Elser  wrote:

>

> What about the other half: "encouraging" users to lag (soon to be) two
> major releases behind?
>


I don't think our current user base needs to be encouraged strongly to
upgrade. And as I said previously I think this change provides them with an
upgrade path that's easier to stomach, but I suspect this is a point we
disagree on.

-- 
Sean

Re: Hadoop 2.0 Support for Accumulo 1.4 Branch

[Josh Elser]

On 11/12/13, 1:26 PM, Sean Busbey wrote:

On Tue, Nov 12, 2013 at 2:48 PM, Josh Elser  wrote:






What about the other half: "encouraging" users to lag (soon to be) two
major releases behind?




I don't think our current user base needs to be encouraged strongly to
upgrade. And as I said previously I think this change provides them with an
upgrade path that's easier to stomach, but I suspect this is a point we
disagree on.



Yes, I believe that disagreement to be accurate :). IMO, HDFS WALs are 
themselves a sufficient feature to strongly encourage users to upgrade 
from 1.4 to 1.5 when running Accumulo in a production environment.


However, I do want to say that my feelings are not solely influenced by 
1.5 features that 1.4 does not have. Providing avenues for users to use 
old versions of code is extremely degrading to the morale of developers 
who are trying to innovate and make things better upstream.


Having to repeatedly answer questions about why things are the way they 
are or why they are sub-optimal with "If you upgraded, it wouldn't be an 
issue" can be very depressing to a growing community. There are multiple 
amazing devs who have invested crazy units of effort into making 1.5 and 
(soon) 1.6 be a reality -- I want to make sure as a community we 
encourage our users to reap the benefits of that work, and let 
aforementioned devs see successes from their effort.


Also, users still aren't precluded from using 1.4 if they so choose (and 
if critical bugs are found, it's very likely for them to be patched in 
1.4). I don't see it as unreasonable to force users to upgrade to a 
newer version of Accumulo to use a newer version of Hadoop.

Re: Hadoop 2.0 Support for Accumulo 1.4 Branch

[Sean Busbey]

On Tue, Nov 12, 2013 at 3:14 PM, William Slacum <
wilhelm.von.cl...@accumulo.net> wrote:

> The language of ACCUMULO-1795 indicated that an acceptable state was
> something that wasn't binary compatible. That's my #1 thing to avoid.
>
>
Ah. So I see, not sure why I phrased that that way. Since the default build
should still be 0.20.203.0, I'm not sure how it'd end up not being binary
compatible. I can update the ticket to clarify the language. Any need to
compile should be limited to running Hadoop 2.2.0.

Sound good?



> > Maybe expressly only doing a binary convenience package for
> > 0.20.203.0?
>
> If we need an extra package, doesn't that mean a user can't just upgrade
> Accumulo?
>

By "binary convenience package" I mean the binary distribution tarball (or
rpms, or whatevs) that we make as a part of the release process. For users
of Hadoop 0.20.203.0, upgrading should be unchanged from how they would
normally get their Accumulo 1.4.x distribution.

ACCUMULO-1796 has some leeway about the convenience packages for people who
want Hadoop 2 support. On the extreme end, they'd have to build from source
and then run a normal upgrade process.

-- 
Sean

Re: Blog for Accumulo?

[Bill Havanki]

Yeah, it doesn't look like planet is very featureful. I don't see tag
support either. Maybe just an old-fashioned blogroll would suffice, at
least initially, or even just stub posts with the first paragraph and a
"Read more" link heading over to the real blog.


On Tue, Nov 12, 2013 at 4:17 PM, Josh Elser  wrote:

> I'm all for having a more "definitive" location which we can use as a
> distribution point for accurate Accumulo information.
>
> We might be able to leverage the ASF's planet instance (
> http://planet.apache.org/) as a aggregation service and link some subset
> of posts to planet on a.a.o? I'm thinking like make a sub-feed of posts to
> planet.apache.org/committers which has an accumulo tag (I'm not sure if
> that's directly possible though).
>
>
> On 11/12/13, 8:43 AM, Bill Havanki wrote:
>
>> Apache hosts blogs for many of its projects. For example:
>>
>> http://blogs.apache.org/hbase/
>>
>> Any interest in establishing one for Accumulo?
>>
>> Bill
>>
>>


-- 
| - - -
| Bill Havanki
| Solutions Architect, Cloudera Government Solutions
| - - -

Re: integration tests

[David Medinets]

With Accumulo v1.5 I ran "mvn clean verify -Daccumulo.it.forkCount=12". The
result was 30 failures in the Testing module. Is this OBE or significant?

Failed tests:   interpreter(org.apache.accumulo.test.ShellServerTest):
HexScan present in root@miniInstance t> interpreter -l(..)
  exporttableImporttable(org.apache.accumulo.test.ShellServerTest):
expected:<0> but was:<1>
  groups(org.apache.accumulo.test.ShellServerTest): expected:<0> but was:<2>
  listscans(org.apache.accumulo.test.ShellServerTest): expected:<0> but
was:<1>
  maxrow(org.apache.accumulo.test.ShellServerTest): expected:<0> but was:<2>
  importDirectory(org.apache.accumulo.test.ShellServerTest): expected:<0>
but was:<3>
  constraint(org.apache.accumulo.test.ShellServerTest): expected:<0> but
was:<3>
  renametable(org.apache.accumulo.test.ShellServerTest): expected:<0> but
was:<3>
  listcompactions(org.apache.accumulo.test.ShellServerTest): expected:<0>
but was:<4>
  classpath(org.apache.accumulo.test.ShellServerTest): expected:<0> but
was:<4>
  du(org.apache.accumulo.test.ShellServerTest): expected:<0> but was:<5>
  grep(org.apache.accumulo.test.ShellServerTest): expected:<0> but was:<6>
  help(org.apache.accumulo.test.ShellServerTest): expected:<0> but was:<6>
  info(org.apache.accumulo.test.ShellServerTest): expected:<0> but was:<6>
  iter(org.apache.accumulo.test.ShellServerTest): expected:<0> but was:<7>
  ping(org.apache.accumulo.test.ShellServerTest): expected:<0> but was:<7>
  user(org.apache.accumulo.test.ShellServerTest): expected:<0> but was:<7>
  debug(org.apache.accumulo.test.ShellServerTest): expected:<0> but was:<7>
  egrep(org.apache.accumulo.test.ShellServerTest): expected:<0> but was:<8>
  merge(org.apache.accumulo.test.ShellServerTest): expected:<0> but was:<9>
  sleep(org.apache.accumulo.test.ShellServerTest): expected:<0> but was:<9>
  trace(org.apache.accumulo.test.ShellServerTest): expected:<0> but was:<9>
  testPertableClasspath(org.apache.accumulo.test.ShellServerTest):
expected:<0> but was:<9>
  setscaniterDeletescaniter(org.apache.accumulo.test.ShellServerTest):
expected:<0> but was:<10>
  clearCls(org.apache.accumulo.test.ShellServerTest): expected:<0> but
was:<10>
  deletemany(org.apache.accumulo.test.ShellServerTest): expected:<10> but
was:<1>
  deleterows(org.apache.accumulo.test.ShellServerTest)
  execfile(org.apache.accumulo.test.ShellServerTest): expected:<0> but
was:<12>
  clonetable(org.apache.accumulo.test.ShellServerTest): expected:<0> but
was:<12>
  notable(org.apache.accumulo.test.ShellServerTest): expected:<0> but
was:<13>

Re: integration tests

[Mike Drob]

Do you get the same failures without the fork count option?
On Nov 12, 2013 8:30 PM, "David Medinets"  wrote:

> With Accumulo v1.5 I ran "mvn clean verify -Daccumulo.it.forkCount=12". The
> result was 30 failures in the Testing module. Is this OBE or significant?
>
> Failed tests:   interpreter(org.apache.accumulo.test.ShellServerTest):
> HexScan present in root@miniInstance t> interpreter -l(..)
>   exporttableImporttable(org.apache.accumulo.test.ShellServerTest):
> expected:<0> but was:<1>
>   groups(org.apache.accumulo.test.ShellServerTest): expected:<0> but
> was:<2>
>   listscans(org.apache.accumulo.test.ShellServerTest): expected:<0> but
> was:<1>
>   maxrow(org.apache.accumulo.test.ShellServerTest): expected:<0> but
> was:<2>
>   importDirectory(org.apache.accumulo.test.ShellServerTest): expected:<0>
> but was:<3>
>   constraint(org.apache.accumulo.test.ShellServerTest): expected:<0> but
> was:<3>
>   renametable(org.apache.accumulo.test.ShellServerTest): expected:<0> but
> was:<3>
>   listcompactions(org.apache.accumulo.test.ShellServerTest): expected:<0>
> but was:<4>
>   classpath(org.apache.accumulo.test.ShellServerTest): expected:<0> but
> was:<4>
>   du(org.apache.accumulo.test.ShellServerTest): expected:<0> but was:<5>
>   grep(org.apache.accumulo.test.ShellServerTest): expected:<0> but was:<6>
>   help(org.apache.accumulo.test.ShellServerTest): expected:<0> but was:<6>
>   info(org.apache.accumulo.test.ShellServerTest): expected:<0> but was:<6>
>   iter(org.apache.accumulo.test.ShellServerTest): expected:<0> but was:<7>
>   ping(org.apache.accumulo.test.ShellServerTest): expected:<0> but was:<7>
>   user(org.apache.accumulo.test.ShellServerTest): expected:<0> but was:<7>
>   debug(org.apache.accumulo.test.ShellServerTest): expected:<0> but was:<7>
>   egrep(org.apache.accumulo.test.ShellServerTest): expected:<0> but was:<8>
>   merge(org.apache.accumulo.test.ShellServerTest): expected:<0> but was:<9>
>   sleep(org.apache.accumulo.test.ShellServerTest): expected:<0> but was:<9>
>   trace(org.apache.accumulo.test.ShellServerTest): expected:<0> but was:<9>
>   testPertableClasspath(org.apache.accumulo.test.ShellServerTest):
> expected:<0> but was:<9>
>   setscaniterDeletescaniter(org.apache.accumulo.test.ShellServerTest):
> expected:<0> but was:<10>
>   clearCls(org.apache.accumulo.test.ShellServerTest): expected:<0> but
> was:<10>
>   deletemany(org.apache.accumulo.test.ShellServerTest): expected:<10> but
> was:<1>
>   deleterows(org.apache.accumulo.test.ShellServerTest)
>   execfile(org.apache.accumulo.test.ShellServerTest): expected:<0> but
> was:<12>
>   clonetable(org.apache.accumulo.test.ShellServerTest): expected:<0> but
> was:<12>
>   notable(org.apache.accumulo.test.ShellServerTest): expected:<0> but
> was:<13>
>

Re: integration tests

[David Medinets]

Great question, Mike. The tests completed successfully in 5 minutes without
the fork count option. Then I tried with varying numbers of fork counts and
eventually got back to 12 forks. And the tests still passed. 
Technological High Jinks. I've run the fork count up to 48 without getting
the tests to fail again. The execution time of the tests does not seem to
change with the fork count.


On Tue, Nov 12, 2013 at 10:45 PM, Mike Drob  wrote:

> Do you get the same failures without the fork count option?
> On Nov 12, 2013 8:30 PM, "David Medinets" 
> wrote:
>
> > With Accumulo v1.5 I ran "mvn clean verify -Daccumulo.it.forkCount=12".
> The
> > result was 30 failures in the Testing module. Is this OBE or significant?
> >
> > Failed tests:   interpreter(org.apache.accumulo.test.ShellServerTest):
> > HexScan present in root@miniInstance t> interpreter -l(..)
> >   exporttableImporttable(org.apache.accumulo.test.ShellServerTest):
> > expected:<0> but was:<1>
> >   groups(org.apache.accumulo.test.ShellServerTest): expected:<0> but
> > was:<2>
> >   listscans(org.apache.accumulo.test.ShellServerTest): expected:<0> but
> > was:<1>
> >   maxrow(org.apache.accumulo.test.ShellServerTest): expected:<0> but
> > was:<2>
> >   importDirectory(org.apache.accumulo.test.ShellServerTest): expected:<0>
> > but was:<3>
> >   constraint(org.apache.accumulo.test.ShellServerTest): expected:<0> but
> > was:<3>
> >   renametable(org.apache.accumulo.test.ShellServerTest): expected:<0> but
> > was:<3>
> >   listcompactions(org.apache.accumulo.test.ShellServerTest): expected:<0>
> > but was:<4>
> >   classpath(org.apache.accumulo.test.ShellServerTest): expected:<0> but
> > was:<4>
> >   du(org.apache.accumulo.test.ShellServerTest): expected:<0> but was:<5>
> >   grep(org.apache.accumulo.test.ShellServerTest): expected:<0> but
> was:<6>
> >   help(org.apache.accumulo.test.ShellServerTest): expected:<0> but
> was:<6>
> >   info(org.apache.accumulo.test.ShellServerTest): expected:<0> but
> was:<6>
> >   iter(org.apache.accumulo.test.ShellServerTest): expected:<0> but
> was:<7>
> >   ping(org.apache.accumulo.test.ShellServerTest): expected:<0> but
> was:<7>
> >   user(org.apache.accumulo.test.ShellServerTest): expected:<0> but
> was:<7>
> >   debug(org.apache.accumulo.test.ShellServerTest): expected:<0> but
> was:<7>
> >   egrep(org.apache.accumulo.test.ShellServerTest): expected:<0> but
> was:<8>
> >   merge(org.apache.accumulo.test.ShellServerTest): expected:<0> but
> was:<9>
> >   sleep(org.apache.accumulo.test.ShellServerTest): expected:<0> but
> was:<9>
> >   trace(org.apache.accumulo.test.ShellServerTest): expected:<0> but
> was:<9>
> >   testPertableClasspath(org.apache.accumulo.test.ShellServerTest):
> > expected:<0> but was:<9>
> >   setscaniterDeletescaniter(org.apache.accumulo.test.ShellServerTest):
> > expected:<0> but was:<10>
> >   clearCls(org.apache.accumulo.test.ShellServerTest): expected:<0> but
> > was:<10>
> >   deletemany(org.apache.accumulo.test.ShellServerTest): expected:<10> but
> > was:<1>
> >   deleterows(org.apache.accumulo.test.ShellServerTest)
> >   execfile(org.apache.accumulo.test.ShellServerTest): expected:<0> but
> > was:<12>
> >   clonetable(org.apache.accumulo.test.ShellServerTest): expected:<0> but
> > was:<12>
> >   notable(org.apache.accumulo.test.ShellServerTest): expected:<0> but
> > was:<13>
> >
>