interesting

[Eric Newton]

Facebook open-sources a distributed logging mechanism:

https://logdevice.io/blog/2018/09/12/open-sourcing-announcement.html


-Eric

Re: Accumulo folks at Hadoop Summit San Jose

[Eric Newton]

I may be in the area that week.  If I am, I'll try go get down there.

On Thu, May 19, 2016 at 3:25 PM, Josh Elser  wrote:

> Try to make the meetup that Billie is setting up and be sure to introduce
> yourself :)
>
> I'll be there this year (if that wasn't obvious by me asking)
>
>
> Claudia Rose wrote:
>
>> I'll be there although I don't know the other "folks" yet.
>>
>> -Original Message-
>> From: Josh Elser [mailto:josh.el...@gmail.com]
>> Sent: Thursday, May 19, 2016 11:01 AM
>> To: dev
>> Cc: u...@accumulo.apache.org
>> Subject: Accumulo folks at Hadoop Summit San Jose
>>
>> Out of curiosity, are there going to be any Accumulo-folks at Hadoop
>> Summit in San Jose, CA at the end of June?
>>
>> - Josh
>>
>>

Re: Monitor Tablet Id mapping

[Eric Newton]

I think there's an option on getsplits (getsplits -v, maybe?).


On Tue, Feb 23, 2016 at 4:31 PM, William Slacum  wrote:

> It's md5sum'd then base64'd. I think if you'd have to build a mapping of
> tablet id <-> md5sum to translate them.
>
> On Tue, Feb 23, 2016 at 3:59 PM,  wrote:
>
> > Anyone know the utility to map the tablet id on the monitor to its actual
> > value? It looks like its Base64 encoded on the monitor, but I don't think
> > that's actually the case.
> >
> >
> >
> >
>

Re: [VOTE] Accumulo 1.6.5-rc1

[Eric Newton]

+1

Signatures for the -bin and -src tarballs check out.
Source release matches tag.
18-hour run of random walk (All.xml) on a small cluster (9 nodes) resulted
in two failures, which I found to be conflicting operations on the same
table in the Concurrent test.

Nice work, and thanks for making the release!

-Eric




On Tue, Feb 9, 2016 at 2:23 PM, Christopher  wrote:

> Accumulo Developers,
>
> Please consider the following candidate for Accumulo 1.6.5.
>
> Git Commit:
> 2263fabd57e765ab14fc47a146b1e2d443e705ca
> Branch:
> 1.6.5-rc1
>
> If this vote passes, a gpg-signed tag will be created using:
> git tag -f -m 'Apache Accumulo 1.6.5' -s rel/1.6.5
> 2263fabd57e765ab14fc47a146b1e2d443e705ca
>
> Staging repo:
> https://repository.apache.org/content/repositories/orgapacheaccumulo-1047
> Source (official release artifact):
>
> https://repository.apache.org/content/repositories/orgapacheaccumulo-1047/org/apache/accumulo/accumulo/1.6.5/accumulo-1.6.5-src.tar.gz
> Binary:
>
> https://repository.apache.org/content/repositories/orgapacheaccumulo-1047/org/apache/accumulo/accumulo/1.6.5/accumulo-1.6.5-bin.tar.gz
> (Append ".sha1", ".md5", or ".asc" to download the signature/hash for a
> given artifact.)
>
> All artifacts were built and staged with:
> mvn release:prepare && mvn release:perform
>
> Signing keys are available at https://www.apache.org/dist/accumulo/KEYS
> (Expected fingerprint: 8CC4F8A2B29C2B040F2B835D6F0CDAE700B6899D)
>
> Release notes (in progress) can be found at:
> https://accumulo.apache.org/release_notes/1.6.5
>
> Please vote one of:
> [ ] +1 - I have verified and accept...
> [ ] +0 - I have reservations, but not strong enough to vote against...
> [ ] -1 - Because..., I do not accept...
> ... these artifacts as the 1.6.5 release of Apache Accumulo.
>
> This vote will end on Fri Feb 12 19:30:00 UTC 2016
> (Fri Feb 12 14:30:00 EST 2016 / Fri Feb 12 11:30:00 PST 2016)
>
> Thanks!
>
> P.S. Hint: download the whole staging repo with
> wget -erobots=off -r -l inf -np -nH \
>
> https://repository.apache.org/content/repositories/orgapacheaccumulo-1047/
> # note the trailing slash is needed
>

Re: [VOTE] Accumulo 1.6.5-rc1

[Eric Newton]

Also just successfully completed a 2B entry continuous ingest test, with
agitation.


On Wed, Feb 10, 2016 at 9:42 AM, Eric Newton <eric.new...@gmail.com> wrote:

> +1
>
> Signatures for the -bin and -src tarballs check out.
> Source release matches tag.
> 18-hour run of random walk (All.xml) on a small cluster (9 nodes) resulted
> in two failures, which I found to be conflicting operations on the same
> table in the Concurrent test.
>
> Nice work, and thanks for making the release!
>
> -Eric
>
>
>
>
> On Tue, Feb 9, 2016 at 2:23 PM, Christopher <ctubb...@apache.org> wrote:
>
>> Accumulo Developers,
>>
>> Please consider the following candidate for Accumulo 1.6.5.
>>
>> Git Commit:
>> 2263fabd57e765ab14fc47a146b1e2d443e705ca
>> Branch:
>> 1.6.5-rc1
>>
>> If this vote passes, a gpg-signed tag will be created using:
>> git tag -f -m 'Apache Accumulo 1.6.5' -s rel/1.6.5
>> 2263fabd57e765ab14fc47a146b1e2d443e705ca
>>
>> Staging repo:
>> https://repository.apache.org/content/repositories/orgapacheaccumulo-1047
>> Source (official release artifact):
>>
>> https://repository.apache.org/content/repositories/orgapacheaccumulo-1047/org/apache/accumulo/accumulo/1.6.5/accumulo-1.6.5-src.tar.gz
>> Binary:
>>
>> https://repository.apache.org/content/repositories/orgapacheaccumulo-1047/org/apache/accumulo/accumulo/1.6.5/accumulo-1.6.5-bin.tar.gz
>> (Append ".sha1", ".md5", or ".asc" to download the signature/hash for a
>> given artifact.)
>>
>> All artifacts were built and staged with:
>> mvn release:prepare && mvn release:perform
>>
>> Signing keys are available at https://www.apache.org/dist/accumulo/KEYS
>> (Expected fingerprint: 8CC4F8A2B29C2B040F2B835D6F0CDAE700B6899D)
>>
>> Release notes (in progress) can be found at:
>> https://accumulo.apache.org/release_notes/1.6.5
>>
>> Please vote one of:
>> [ ] +1 - I have verified and accept...
>> [ ] +0 - I have reservations, but not strong enough to vote against...
>> [ ] -1 - Because..., I do not accept...
>> ... these artifacts as the 1.6.5 release of Apache Accumulo.
>>
>> This vote will end on Fri Feb 12 19:30:00 UTC 2016
>> (Fri Feb 12 14:30:00 EST 2016 / Fri Feb 12 11:30:00 PST 2016)
>>
>> Thanks!
>>
>> P.S. Hint: download the whole staging repo with
>> wget -erobots=off -r -l inf -np -nH \
>>
>> https://repository.apache.org/content/repositories/orgapacheaccumulo-1047/
>> # note the trailing slash is needed
>>
>
>

Re: [VOTE] Accumulo 1.6.5-rc1

[Eric Newton]

I've run all the unit and integration tests against hadoop 2.2.0.

I'm going to switch to the 1.7 branch to try and get some testing done
ahead of the next RC candidate.


On Wed, Feb 10, 2016 at 12:34 PM, Eric Newton <eric.new...@gmail.com> wrote:

> Also just successfully completed a 2B entry continuous ingest test, with
> agitation.
>
>
> On Wed, Feb 10, 2016 at 9:42 AM, Eric Newton <eric.new...@gmail.com>
> wrote:
>
>> +1
>>
>> Signatures for the -bin and -src tarballs check out.
>> Source release matches tag.
>> 18-hour run of random walk (All.xml) on a small cluster (9 nodes)
>> resulted in two failures, which I found to be conflicting operations on the
>> same table in the Concurrent test.
>>
>> Nice work, and thanks for making the release!
>>
>> -Eric
>>
>>
>>
>>
>> On Tue, Feb 9, 2016 at 2:23 PM, Christopher <ctubb...@apache.org> wrote:
>>
>>> Accumulo Developers,
>>>
>>> Please consider the following candidate for Accumulo 1.6.5.
>>>
>>> Git Commit:
>>> 2263fabd57e765ab14fc47a146b1e2d443e705ca
>>> Branch:
>>> 1.6.5-rc1
>>>
>>> If this vote passes, a gpg-signed tag will be created using:
>>> git tag -f -m 'Apache Accumulo 1.6.5' -s rel/1.6.5
>>> 2263fabd57e765ab14fc47a146b1e2d443e705ca
>>>
>>> Staging repo:
>>> https://repository.apache.org/content/repositories/orgapacheaccumulo-1047
>>> Source (official release artifact):
>>>
>>> https://repository.apache.org/content/repositories/orgapacheaccumulo-1047/org/apache/accumulo/accumulo/1.6.5/accumulo-1.6.5-src.tar.gz
>>> Binary:
>>>
>>> https://repository.apache.org/content/repositories/orgapacheaccumulo-1047/org/apache/accumulo/accumulo/1.6.5/accumulo-1.6.5-bin.tar.gz
>>> (Append ".sha1", ".md5", or ".asc" to download the signature/hash for a
>>> given artifact.)
>>>
>>> All artifacts were built and staged with:
>>> mvn release:prepare && mvn release:perform
>>>
>>> Signing keys are available at https://www.apache.org/dist/accumulo/KEYS
>>> (Expected fingerprint: 8CC4F8A2B29C2B040F2B835D6F0CDAE700B6899D)
>>>
>>> Release notes (in progress) can be found at:
>>> https://accumulo.apache.org/release_notes/1.6.5
>>>
>>> Please vote one of:
>>> [ ] +1 - I have verified and accept...
>>> [ ] +0 - I have reservations, but not strong enough to vote against...
>>> [ ] -1 - Because..., I do not accept...
>>> ... these artifacts as the 1.6.5 release of Apache Accumulo.
>>>
>>> This vote will end on Fri Feb 12 19:30:00 UTC 2016
>>> (Fri Feb 12 14:30:00 EST 2016 / Fri Feb 12 11:30:00 PST 2016)
>>>
>>> Thanks!
>>>
>>> P.S. Hint: download the whole staging repo with
>>> wget -erobots=off -r -l inf -np -nH \
>>>
>>>
>>> https://repository.apache.org/content/repositories/orgapacheaccumulo-1047/
>>> # note the trailing slash is needed
>>>
>>
>>
>

Re: could not start the service

[Eric Newton]

Hmm, the configuration files for accumulo are conf/accumulo-site.xml, and
conf/accumulo-env.sh, and of course the host files for the various roles.

The basic accumulo start scripts are not looking for "accumulo-config.xml".



On Tue, Feb 2, 2016 at 9:38 AM, gunjanindia  wrote:

> While running accumulo init command i am getting the following error
> could be some error in accumulo-config.xml
> [accumulo@lumify ~]$ /usr/lib/accumulo/bin/accumulo init
> *Could not infer a Monitor role. You need to either define the MONITOR env
> variable,* define "/usr/lib/accumulo/conf/monitor", or make sure
> "/usr/lib/accumulo/conf/masters" is non-empty.
> please guide me
>
> regards
> Gunjan Kumar
>
>
>
> --
> View this message in context:
> http://apache-accumulo.1065345.n5.nabble.com/could-not-start-the-service-tp16000.html
> Sent from the Developers mailing list archive at Nabble.com.
>

Re: question on data block cache

[Eric Newton]

By default, the files used to store the data are compressed.  The block
cache stores the data in uncompressed blocks. So, even if your stored data
is <16G/tserver, the amount it might take in memory could be much larger.

There might be bugs in computing this number, too. :-)

However, I see lots of systems running with a 100% data block cache hit
rate.

-Eric

On Fri, Jan 22, 2016 at 9:49 AM, z11373  wrote:

> Thanks Eric! Yes, the data block cache is already enabled. From the table
> config, I see:
>
> default| table.cache.block.enable .. | false
> site   |@override .. | true
>
> Actually, there is no ingestion task running at that time when I ran the
> tests (which only do read).
> What is other possibility? Looking at the total tablets and total entries
> being queried from the Monitor UI, they are not like big number, so I have
> some doubt that 16GB data cache is not enough space, but I could be wrong.
>
> Thanks,
> Z
>
>
>
> --
> View this message in context:
> http://apache-accumulo.1065345.n5.nabble.com/question-on-data-block-cache-tp15906p15925.html
> Sent from the Developers mailing list archive at Nabble.com.
>

Re: question on data block cache

[Eric Newton]

There's an easy way, but it's not exactly fast.

Change the compression type on the table and compact it.  Then the numbers
provided by the "df" in the shell will be very reliable for the
uncompressed size of the data.

-Eric



On Fri, Jan 22, 2016 at 10:55 AM, z11373  wrote:

> Is there a quick and sure way to find this out?
> I just want to confirm if that is the case, can't really tell from the data
> shown on Monitor UI.
>
> Thanks,
> Z
>
>
>
> --
> View this message in context:
> http://apache-accumulo.1065345.n5.nabble.com/question-on-data-block-cache-tp15906p15927.html
> Sent from the Developers mailing list archive at Nabble.com.
>

Re: question on data block cache

[Eric Newton]

There are some debug-level cache statistics printed to the tserver logs
periodically. It will take some digging around to figure out what they
mean.  They might help.

Re: question on data block cache

[Eric Newton]

Assuming you have on-going ingest, accumulo is going to re-sort/organize
your data. The new files that result will cause cache hit failures.

Make sure you enable data caching for your table since it is not turned on
for user tables by default.

-Eric


On Wed, Jan 20, 2016 at 11:22 AM, z11373  wrote:

> Hi,
> I see the data cache hit rate percentage is in 80s.
> My client app sends a lot of small scan requests, and looking at per tablet
> details, I see the same number of tablets (where the data being queried
> from). We set the data cache size to 16GB, so I'd think it should have
> plenty of room. Is there a way for me to see if the percentage doesn't get
> higher (would expect in 90s if not 100%) because of 16GB not big enough, or
> it's just there are many data/tablets to load from?
>
> Thanks,
> Z
>
>
>
> --
> View this message in context:
> http://apache-accumulo.1065345.n5.nabble.com/question-on-data-block-cache-tp15906.html
> Sent from the Developers mailing list archive at Nabble.com.
>

Re: tablet split

[Eric Newton]

Accumulo will balance the tablets based on the configured balancer.

Without getting down into the details, the splits will be moved to other
nodes.

The Details:

It depends. With the default balancer, it will try to smooth out the number
of tablets among servers, by table. So, if this table goes from 1 tablet to
3, and there are at least 3 servers, each split will eventually find itself
moved to separate server. But, if you add one split among hundreds, it may
not make much of a difference to bother moving the tablet.

-Eric


On Tue, Oct 20, 2015 at 10:39 AM, z11373  wrote:

> As my understanding, Accumulo will have data already sorted with row id,
> and
> if the number of rows is growing, it will split the tablet at one point.
> For example, let say I have following row ids:
>
> 1_abcxxx
> 1_abdxxx
> 1_abexxx
> 1_abfxxx
> 1_abgxxx
> 1_abhxxx
> 1_abixxx
> ...
> 1_zzzxxx
> 2_abcxxx
> 2_abdxxx
> 2_abexxx
> 2_abfxxx
> 2_abgxxx
> 2_abhxxx
> ...
>
> Let say the data with row id starts with "1_" has a million of rows, and
> for
> sake of example, let say the tablet size is 400K, so in this case the "1_"
> data will be split into 3 tablets.
>
> My question is will Accumulo distribute those 3 tablets into different
> tablet server nodes? Or perhaps two or all of them will remain in that
> original tablet server?
>
>
> Thanks,
> Z
>
>
>
>
> --
> View this message in context:
> http://apache-accumulo.1065345.n5.nabble.com/tablet-split-tp15399.html
> Sent from the Developers mailing list archive at Nabble.com.
>

Re: [RESULT] [VOTE] Accumulo 1.6.4-rc1

[Eric Newton]

Keith kicked off a Continuous Ingest test of 1.6.4 on a cluster of 17
tablet servers.  No servers were killed (intentionally, or otherwise). No
data was lost (37B Key/Value pairs).

-Eric

On Fri, Oct 2, 2015 at 10:14 AM, Josh Elser  wrote:

> This vote passes with 4 +1s and nothing else.
>
> Thanks to all who took the time to evaluate the RC.
>
> - Josh
>
> Josh Elser wrote:
>
>> Accumulo Developers,
>>
>> Please consider the following candidate for Accumulo 1.6.4.
>>
>> Git Commit:
>> edba4f4ca95d9e8ec0299a631234e5c9a319f9ec
>> Branch:
>> 1.6.4-rc1
>>
>> If this vote passes, a gpg-signed tag will be created using:
>> git tag -f -m 'Apache Accumulo 1.6.4' -s 1.6.4
>> edba4f4ca95d9e8ec0299a631234e5c9a319f9ec
>>
>> Staging repo:
>> https://repository.apache.org/content/repositories/orgapacheaccumulo-1043
>> Source (official release artifact):
>>
>> https://repository.apache.org/content/repositories/orgapacheaccumulo-1043/org/apache/accumulo/accumulo/1.6.4/accumulo-1.6.4-src.tar.gz
>>
>> Binary:
>>
>> https://repository.apache.org/content/repositories/orgapacheaccumulo-1043/org/apache/accumulo/accumulo/1.6.4/accumulo-1.6.4-bin.tar.gz
>>
>> (Append ".sha1", ".md5", or ".asc" to download the signature/hash for a
>> given artifact.)
>>
>> All artifacts were built and staged with:
>> mvn release:prepare && mvn release:perform
>>
>> Signing keys are available at https://www.apache.org/dist/accumulo/KEYS
>> (Expected fingerprint: ABC8914C675FAD3FA74F39B2D146D62CAB471AE9)
>>
>> Release notes have not yet been started, but the primary purpose for
>> this release is to provide a 1.6 release for the bulk-load dataloss bug.
>>
>> Please vote one of:
>> [ ] +1 - I have verified and accept...
>> [ ] +0 - I have reservations, but not strong enough to vote against...
>> [ ] -1 - Because..., I do not accept...
>> ... these artifacts as the 1.6.4 release of Apache Accumulo.
>>
>> This vote will end on Fri Oct 2 04:30:00 UTC 2015
>> (Fri Oct 2 00:30:00 EDT 2015 / Thu Oct 1 21:30:00 PDT 2015)
>>
>> Thanks!
>>
>> P.S. Hint: download the whole staging repo with
>> wget -erobots=off -r -l inf -np -nH \
>>
>>
>> https://repository.apache.org/content/repositories/orgapacheaccumulo-orgapacheaccumulo-1043/
>>
>> # note the trailing slash is needed
>>
>

Re: [VOTE] Accumulo 1.6.4-rc1

[Eric Newton]

Thanks for taking this on, Josh.

I've verified the sources match the 1.6 branch.
The signature and md5 matches the source tarball.

I ran the following basic tests:

$ mvn -Pthrift,assemble,docs,sunny verify

I reviewed the differences between 1.6.3 and 1.6.4-rc1.

+1




On Tue, Sep 29, 2015 at 12:07 AM, Josh Elser  wrote:

> Accumulo Developers,
>
> Please consider the following candidate for Accumulo 1.6.4.
>
> Git Commit:
> edba4f4ca95d9e8ec0299a631234e5c9a319f9ec
> Branch:
> 1.6.4-rc1
>
> If this vote passes, a gpg-signed tag will be created using:
> git tag -f -m 'Apache Accumulo 1.6.4' -s 1.6.4
> edba4f4ca95d9e8ec0299a631234e5c9a319f9ec
>
> Staging repo:
> https://repository.apache.org/content/repositories/orgapacheaccumulo-1043
> Source (official release artifact):
> https://repository.apache.org/content/repositories/orgapacheaccumulo-1043/org/apache/accumulo/accumulo/1.6.4/accumulo-1.6.4-src.tar.gz
> Binary:
> https://repository.apache.org/content/repositories/orgapacheaccumulo-1043/org/apache/accumulo/accumulo/1.6.4/accumulo-1.6.4-bin.tar.gz
> (Append ".sha1", ".md5", or ".asc" to download the signature/hash for a
> given artifact.)
>
> All artifacts were built and staged with:
> mvn release:prepare && mvn release:perform
>
> Signing keys are available at https://www.apache.org/dist/accumulo/KEYS
> (Expected fingerprint: ABC8914C675FAD3FA74F39B2D146D62CAB471AE9)
>
> Release notes have not yet been started, but the primary purpose for this
> release is to provide a 1.6 release for the bulk-load dataloss bug.
>
> Please vote one of:
> [ ] +1 - I have verified and accept...
> [ ] +0 - I have reservations, but not strong enough to vote against...
> [ ] -1 - Because..., I do not accept...
> ... these artifacts as the 1.6.4 release of Apache Accumulo.
>
> This vote will end on Fri Oct  2 04:30:00 UTC 2015
> (Fri Oct  2 00:30:00 EDT 2015 / Thu Oct  1 21:30:00 PDT 2015)
>
> Thanks!
>
> P.S. Hint: download the whole staging repo with
> wget -erobots=off -r -l inf -np -nH \
>
>
> https://repository.apache.org/content/repositories/orgapacheaccumulo-orgapacheaccumulo-1043/
> # note the trailing slash is needed
>

Re: [ANNOUNCE] Apache Gora 0.6.1 Release

[Eric Newton]

I pulled down their git repo and upgraded it to 1.7.0.

There's code in there (getPartitions) that uses the tablet locator and
KeyExtent.

I wonder if there's a better way to do what it wants (with
AccumuloInputFormat?).

Keith, do you have any ideas on where this code originated?

It should be updated to use the ClientContext stuff.

-Eric

On Tue, Sep 15, 2015 at 11:12 AM, Josh Elser  wrote:

> Oy, we need to update their dependency...
>
> Also, another project which would be good to add to Christopher's
> rc-verify.
>
>  Original Message 
> Subject: [ANNOUNCE] Apache Gora 0.6.1 Release
> Date: Mon, 14 Sep 2015 23:26:38 -0700
> From: lewis john mcgibbney 
> Reply-To: u...@hbase.apache.org
> To: u...@hbase.apache.org ,
> u...@accumulo.apache.org,  u...@cassandra.apache.org,
> solr-u...@lucene.apache.org,  u...@giraph.apache.org, u...@avro.apache.org
> ,   ,
> u...@nutch.apache.org , annou...@apache.org,
> u...@spark.apache.org, u...@camel.apache.org
>
> Hi All,
>
> The Apache Gora team are pleased to announce the immediate availability of
> Apache Gora 0.6.1.
>
> What is Gora?
> Gora is a framework which provides an in-memory data model and persistence
> for big data. Gora supports persisting to column stores, key value stores,
> document stores and RDBMSs, and analyzing the data with extensive Apache
> Hadoop™  MapReduce
>  support. This
> release also offers input and output formats for Apache Spark.
>
> Whats in this release?
>
> This release addresses a modest 21 issues  with
> many improvements and bug fixes for the gora-mongodb
>  module, resolution of a
> major bug whilst flushing data to Apache Solr, a gora-gradle plugin
>  and our Gora Spark
> backend
> support . Drop by our
> mailing lists and ask questions for information on any of the above.
>
> We provide Gora support for the following projects
>
>- Apache Avro 1.7.6
>- Apache Hadoop 1.2.1 and 2.5.2
>- Apache HBase 0.98.8-hadoop2 (although also tested with 1.X)
>- Apache Cassandra 2.0.2
>- Apache Solr 4.10.3
>- MongoDB 2.6.X
>- Apache Accumlo 1.5.1
>- Apache Spark 1.4.1
>
> Gora is released as both source code, downloads for which can be found at
> our downloads page  as well as
> Maven
> artifacts which can be found on Maven central
> .
>
> Thank you
> Lewis
> (On behalf of the Apache Gora PMC)
>
> http://people.apache.org/~lewismc || @hectorMcSpector ||
> http://www.linkedin.com/in/lmcgibbney
>
>   Apache Gora V.P || Apache Nutch PMC || Apache Any23 V.P ||
> Apache OODT PMC
>Apache Open Climate Workbench PMC || Apache Tika PMC || Apache
> TAC
> Apache Usergrid || Apache HTrace (incubating) || Apache CommonsRDF
> (incubating)
>
>

Re: [VOTE] Accumulo 1.5.4-rc2

[Eric Newton]

Still +1

On Thu, Sep 17, 2015 at 6:34 PM, Josh Elser  wrote:

> Relatively quiet on this RC so far.
>
> ~1 day left on this one. Make sure to double check the licenses (that's
> all that's changed here over rc1) and cast your vote.
>
> Thanks.
>
>
> Josh Elser wrote:
>
>> Accumulo Developers,
>>
>> Please consider the following candidate for Accumulo 1.5.4.
>>
>> Git Commit:
>> 151db23e7d95cf77c08023ee18b7e524f78286fc
>> Branch:
>> 1.5.4-rc2
>>
>> If this vote passes, a gpg-signed tag will be created using:
>> git tag -f -m 'Apache Accumulo 1.5.4' -s 1.5.4
>> 151db23e7d95cf77c08023ee18b7e524f78286fc
>>
>> Staging repo:
>> https://repository.apache.org/content/repositories/orgapacheaccumulo-1041
>> Source (official release artifact):
>>
>> https://repository.apache.org/content/repositories/orgapacheaccumulo-1041/org/apache/accumulo/accumulo/1.5.4/accumulo-1.5.4-src.tar.gz
>>
>> Binary:
>>
>> https://repository.apache.org/content/repositories/orgapacheaccumulo-1041/org/apache/accumulo/accumulo/1.5.4/accumulo-1.5.4-bin.tar.gz
>>
>> (Append ".sha1", ".md5", or ".asc" to download the signature/hash for a
>> given artifact.)
>>
>> All artifacts were built and staged with:
>> mvn release:prepare && mvn release:perform
>>
>> Signing keys are available at https://www.apache.org/dist/accumulo/KEYS
>> (Expected fingerprint: ABC8914C675FAD3FA74F39B2D146D62CAB471AE9)
>>
>> Release notes (in progress) can be found at
>> https://accumulo.apache.org/release_notes/1.5.4
>>
>> Please vote one of:
>> [ ] +1 - I have verified and accept...
>> [ ] +0 - I have reservations, but not strong enough to vote against...
>> [ ] -1 - Because..., I do not accept...
>> ... these artifacts as the 1.5.4 release of Apache Accumulo.
>>
>> This vote will end on Fri Sep 18 22:00:00 UTC 2015
>> (Fri Sep 18 18:00:00 EDT 2015 / Fri Sep 18 15:00:00 PDT 2015)
>>
>> Thanks!
>>
>> P.S. Hint: download the whole staging repo with
>> wget -erobots=off -r -l inf -np -nH \
>>
>> https://repository.apache.org/content/repositories/orgapacheaccumulo-1041/
>> # note the trailing slash is needed
>>
>

Re: [VOTE] Accumulo 1.5.4-rc1

[Eric Newton]

+1

Verified signatures, built, ran unit tests, started everything up and
performed (limited) tests.

Thanks Josh!

-Eric


On Tue, Sep 8, 2015 at 2:39 PM, Josh Elser  wrote:

> Also, a heads-up since I had one question about this already: you
> (hopefully) will notice that this was signed using a different key than
> previously for me. This is expected.
>
> I built this release on a virtual server (under my virtual but not
> physical control). As such, I did not feel comfortable placing my existing
> private key on this server and created a new one instead.
>
> You'll want to recv-keys on my new key when verifying checksums: `gpg
> --keyserver pgp.mit.edu --recv-keys AB471AE9`
>
>
> Josh Elser wrote:
>
>> Accumulo Developers,
>>
>> Please consider the following candidate for Accumulo 1.5.4.
>>
>> Git Commit:
>> 12a1041dcbb7f3b10543c305f27ece4b0d65ab9c
>> Branch:
>> 1.5.4-rc1
>>
>> If this vote passes, a gpg-signed tag will be created using:
>> git tag -f -m 'Apache Accumulo 1.5.4' -s 1.5.4
>> 12a1041dcbb7f3b10543c305f27ece4b0d65ab9c
>>
>> Staging repo:
>> https://repository.apache.org/content/repositories/orgapacheaccumulo-1039
>> Source (official release artifact):
>>
>> https://repository.apache.org/content/repositories/orgapacheaccumulo-1039/org/apache/accumulo/accumulo/1.5.4/accumulo-1.5.4-src.tar.gz
>>
>> Binary:
>>
>> https://repository.apache.org/content/repositories/orgapacheaccumulo-1039/org/apache/accumulo/accumulo/1.5.4/accumulo-1.5.4-bin.tar.gz
>>
>> (Append ".sha1", ".md5", or ".asc" to download the signature/hash for a
>> given artifact.)
>>
>> All artifacts were built and staged with:
>> mvn release:prepare && mvn release:perform
>>
>> Signing keys are available at https://www.apache.org/dist/accumulo/KEYS
>> (Expected fingerprint: ABC8914C675FAD3FA74F39B2D146D62CAB471AE9)
>>
>> Release notes (in progress) can be found at
>> https://accumulo.apache.org/release_notes/1.5.4
>>
>> Please vote one of:
>> [ ] +1 - I have verified and accept...
>> [ ] +0 - I have reservations, but not strong enough to vote against...
>> [ ] -1 - Because..., I do not accept...
>> ... these artifacts as the 1.5.4 release of Apache Accumulo.
>>
>> This vote will end on Thurs Sep 10 23:00:00 UTC 2015
>> (Thurs Sep 10 20:00:00 EDT 2015 / Thurs Sep 10 17:00:00 PDT 2015)
>>
>> Thanks!
>>
>> P.S. Hint: download the whole staging repo with
>> wget -erobots=off -r -l inf -np -nH \
>>
>> https://repository.apache.org/content/repositories/orgapacheaccumulo-1039/
>> # note the trailing slash is needed
>>
>

Re: exception thrown during minor compaction

[Eric Newton]

Unrelated to the error you are experiencing:

You can add a Constraint that will reject inserts with values other than 1
or -1.

-Eric



On Tue, Sep 1, 2015 at 3:24 PM, z11373  wrote:

> Hi Josh,
> I figured it out that it's actually a human bug :-)
> There is existing code somewhere which I overlooked earlier, it wrote to
> the
> table with empty value, hence makes sense it's giving that error.
>
> I'll try to enforce the semantic from our use case, delete the row
> shouldn't
> happen, and instead it should insert with -1 as value, so I should be good
> here. I also tested deleting a single row, without re-insert, and that row
> not returned from running scan command, which is correct. I didn't try
> re-add that row, which I guess would hit that bug you guys mentioned.
>
> I have one more question, is it possible to keep the row with value is
> greater than 1? For example, let say I insert:
> ['foo', 1]
> ['foo', 1]
> ['bar', 1]
> ['foo', 1]
>
> The scan currently returns:
> ['bar', 1]
> ['foo', 3]
>
> I want it only returns:
> ['foo', 3]
>
> So basically, during compaction it doesn't include those with value = 1 to
> new files, hence this will also make the stats table small since it has so
> many rows with value = 1 (which we don't care). Hopefully the solution also
> can be used for value is other than 1 (i.e. keep rows with value > 10)
> It looks like I may be able to use the built-in RegEx filter for that, but
> perhaps there is better way?
>
> Oh, another thing that I observed during my testing:
>
> root@dev > setiter -t combiner2 -p 15 -scan -minc -majc -n sumcombiners
> -class org.apache.accumulo.core.iterators.user.SummingCombiner
> SummingCombiner interprets Values as Longs and adds them together.  A
> variety of encodings (variable length, fixed length, or string) are
> available
> --> set SummingCombiner parameter all, set to true to apply
> Combiner
> to every column, otherwise leave blank. if true, columns option will be
> ignored.: true
> --> set SummingCombiner parameter columns, [: qual>]{,[:]} escape non-alphanum chars using %.:
> --> set SummingCombiner parameter lossy, if true, failed decodes
> are
> ignored. Otherwise combiner will error on failed decodes (default false):
> :
> --> set SummingCombiner parameter type,
> :
> org.apache.accumulo.core.client.lexicoder.LongLexicoder
> 2015-09-01 15:22:46,587 [shell.Shell] ERROR:
> java.lang.IllegalArgumentException: bad encoder option
>
> Do you know why I got that error?
> 'org.apache.accumulo.core.client.lexicoder.LongLexicoder' should be the
> correct full class name.
>
>
> Thanks a lot,
> Z
>
>
>
> --
> View this message in context:
> http://apache-accumulo.1065345.n5.nabble.com/exception-thrown-during-minor-compaction-tp15010p15030.html
> Sent from the Developers mailing list archive at Nabble.com.
>

Re: exception thrown during minor compaction

[Eric Newton]

You may be seeing a delete marker.  In a compaction, you will see delete
markers, which have an empty value. You will have to check the delete flag
on the key before grabbing the Value.

-Eric

On Mon, Aug 31, 2015 at 8:28 PM, z11373  wrote:

> Thanks Josh! I am going to do more experiments, because this is really
> weird.
> I'll post the update if there are any interesting stuff I found out later.
>
> Thanks,
> Z
>
>
>
> --
> View this message in context:
> http://apache-accumulo.1065345.n5.nabble.com/exception-thrown-during-minor-compaction-tp15010p15017.html
> Sent from the Developers mailing list archive at Nabble.com.
>

Re: exception thrown during minor compaction

[Eric Newton]

Sure, but not during a compaction that does not involve all the underlying
files. The delete keys must be propagated.

I'm not completely familiar with the underlying libraries that help you
write iterators, I just know it's a common mistake.


On Mon, Aug 31, 2015 at 11:53 PM, Josh Elser <josh.el...@gmail.com> wrote:

> Shouldn't the delete be masked at a lower layer (DeletingIterator)? Or am
> I forgetting that Combiners see that value somehow (and maybe
> SummingCombiner is broken)?
>
>
> Eric Newton wrote:
>
>> You may be seeing a delete marker.  In a compaction, you will see delete
>> markers, which have an empty value. You will have to check the delete flag
>> on the key before grabbing the Value.
>>
>> -Eric
>>
>> On Mon, Aug 31, 2015 at 8:28 PM, z11373<z11...@outlook.com>  wrote:
>>
>> Thanks Josh! I am going to do more experiments, because this is really
>>> weird.
>>> I'll post the update if there are any interesting stuff I found out
>>> later.
>>>
>>> Thanks,
>>> Z
>>>
>>>
>>>
>>> --
>>> View this message in context:
>>>
>>> http://apache-accumulo.1065345.n5.nabble.com/exception-thrown-during-minor-compaction-tp15010p15017.html
>>> Sent from the Developers mailing list archive at Nabble.com.
>>>
>>>
>>

Re: HDFS-8722

[Eric Newton]

I just started using 2.7.1, but I've not done any performance testing, yet.

On Thu, Aug 27, 2015 at 12:01 PM, Josh Elser josh.el...@gmail.com wrote:

 https://issues.apache.org/jira/browse/HDFS-8722

 Curious if anyone else has run into this one. Saw this one brought up over
 in an HBase JIRA issue. Seems to imply that write speed might be degraded
 by our WAL use in 2.7.1. Not sure if we should advertise any warning as I
 don't have a feeling yet for if we notice an ingest impact.

 - Joshd

Re: Release Happy Hour

[Eric Newton]

It looks like the best time is July 28th at 5PM. I'll re-arrange my
schedule so I can make it, too.

I suggest Frisco's in Columbia.

Maybe we can get Aaron, Billie and Mike to sign Accumulo books?

-Eric

On Wed, Jul 15, 2015 at 4:13 PM, Eric Newton eric.new...@gmail.com wrote:

 I've been suggesting a happy hour to celebrate the recent releases.

 Here's a doodle to coordinate when:

 http://doodle.com/37qx4v5e6ay5hxy9

 See you soon! If you aren't in the central Maryland area, I'll bring you
 in with a Google hangout. :-)

 -Eric

Release Happy Hour

[Eric Newton]

I've been suggesting a happy hour to celebrate the recent releases.

Here's a doodle to coordinate when:

http://doodle.com/37qx4v5e6ay5hxy9

See you soon! If you aren't in the central Maryland area, I'll bring you in
with a Google hangout. :-)

-Eric

Re: Accumulo debug mode possible?

[Eric Newton]

You can add flags to the jvm arguments in accumulo-env.sh which will allow
remote debugging.

But we almost never do that. We generally write unit tests, and debug those.

The primary way we debug is with logging, and jstack'ing the running
processes.
When debugging a test system, we'll preserve the write-ahead logs which can
be used to determine what happened, especially for updates to the metadata
tables.

We have taken memory dumps of running programs.  This usually kills the
process because the act of dumping memory stops the process too long to
maintain zookeeper sessions.

There's also the monitor, which will highlight unusual conditions, HDFS
errors, show hot-spotting.

For the most part, it's log analysis, and usually a post-mortem.

-Eric

On Wed, Jul 15, 2015 at 5:04 PM, Leon Xu leon.guodong...@gmail.com wrote:

 Hi Accumulo devs,

 I am wondering if there is a way to start accumulo in a debug mode. So that
 you can connect your code with the server (either master or tserver) to do
 single step debugging?
 And if not, how do you normally develop and debug to fix certain issues?


 Thanks
 Leon

Re: Post 1.5.3 and 1.6.3

[Eric Newton]

 This started making me think about making this clear for other components
 like FATE and RandomWalk.


Accumulo dev's have made some effort to extract Fate, Tracing, RandomWalk
and even ContinuousIngest, all to see the code copied (tracing, CI), with
little attribution, or ignored (RW, Fate). The effort to create independent
projects has largely been wasted.  Worse, it has led to convoluted
duplication within our own code base. For example, multiple copies of
zookeeper-related code.

I would rather do something like port RW to HBase, and adopt that version
for Accumulo, like we have for tracing.  Otherwise, I wouldn't bother
putting any effort into separating these modules with some abstract notion
that someone else might use it.

-Eric


On Wed, Jul 8, 2015 at 3:43 PM, Josh Elser josh.el...@gmail.com wrote:

 Some thoughts myself...

 John Vines brought up to me privately the topic of separating out the
 RFile code from core. This started making me think about making this clear
 for other components like FATE and RandomWalk. These all have some level of
 separation, but they often get other things dropped into the same bucket
 (e.g. ZK-retry code in FATE, Accumulo implementation classes in
 RandomWalk). Maybe there are more things we could do.

 It would be nice to start trying to pull out these frameworks/sub-projects
 into discrete packages. I think it would help us with testing and proper
 separation of logic. Long term, maybe other projects would see the value
 and consider using/adopting them and grow into their own
 separately-versioned artifacts. It would be nice to start these efforts now
 to eventually reap the benefits.


 2.0 and the new client API is a little scary now that we get another tick
 closer to it. I know it's been Christopher's brain-child so far (which is
 fine -- not meant to be taken in a negative context), but, if we really do
 want to adopt it, we should make a concerted effort to start integrating
 and reviewing it. Given how far away this seems, 1.8 and 1.9 could happen
 (or client API just gets targeted for a 3.0 -- numbers are just numbers).


 We have some decent script improvements for 1.8 already in (PID files
 _finally_). Would be nice to clean up the rest of the scripts too (notably
 the stop scripts need some love).


 Some other back-burner thoughts: better client API metrics, more
 server-side tracing instrumentation, Adam's iterator-stack collapsing perf
 ticket, keep tabs on HDFS tracing impl, keep tabs on HTrace's GUI work,
 finish the Accumulo monitor rewrite (aka REST server + servlet3).

 - Josh


 Josh Elser wrote:

 Thanks to the efforts spearheaded by Christopher and verified by
 everyone else, we now have 1.5.3 and 1.6.3 releases!

 To keep the ball rolling, what's next? High level questions that come to
 mind...

 * When do we do 1.7.1 and/or 1.8.0?
 * What bug-fixes do we have outstanding for 1.7.1?
 * What other minor improvements do people want for 1.8.0?
 * Where does 2.0.0 stand? Should we make a bigger effort to getting the
 new client API stuff Christopher had started into Apache?

 Feel free to brainstorm here and/or on JIRA (tagging relevant issues to
 the desired fixVersion)

 - Josh

happy test

[Eric Newton]

[insert Dilbert awkward happy dance comic]

This is the result of my most recent map-reduce test run on 10-node amazon
cluster.

Thanks to Josh for working on test stability with me. Thank you for
everyone who has contributed tests that let me make sweeping changes with
some confidence.

PASS
   org.apache.accumulo.test.UsersIT

org.apache.accumulo.test.replication.GarbageCollectorCommunicatesWithTServersIT

org.apache.accumulo.test.replication.UnusedWalDoesntCloseReplicationStatusIT
   org.apache.accumulo.test.replication.UnorderedWorkAssignerReplicationIT
   org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT
   org.apache.accumulo.test.replication.MultiInstanceReplicationIT
   org.apache.accumulo.test.functional.TabletStateChangeIteratorIT
   org.apache.accumulo.test.functional.BalanceAfterCommsFailureIT
   org.apache.accumulo.test.replication.MultiTserverReplicationIT
   org.apache.accumulo.test.performance.metadata.FastBulkImportIT
   org.apache.accumulo.test.functional.DeleteTableDuringSplitIT
   org.apache.accumulo.test.functional.SimpleBalancerFairnessIT
   org.apache.accumulo.test.functional.RecoveryWithEmptyRFileIT
   org.apache.accumulo.test.replication.ReplicationRandomWalkIT
   org.apache.accumulo.test.functional.ConfigurableCompactionIT
   org.apache.accumulo.test.server.security.SystemCredentialsIT
   org.apache.accumulo.test.functional.DeletedTablesDontFlushIT
   org.apache.accumulo.test.MissingWalHeaderCompletesRecoveryIT
   org.apache.accumulo.test.functional.BulkSplitOptimizationIT
   org.apache.accumulo.test.replication.KerberosReplicationIT
   org.apache.accumulo.test.TracerRecoversAfterOfflineTableIT
   org.apache.accumulo.test.functional.AccumuloInputFormatIT
   org.apache.accumulo.test.performance.RollWALPerformanceIT
   org.apache.accumulo.test.replication.CyclicReplicationIT
   org.apache.accumulo.test.functional.DynamicThreadPoolsIT
   org.apache.accumulo.test.functional.CreateManyScannersIT
   org.apache.accumulo.test.RecoveryCompactionsAreFlushesIT
   org.apache.accumulo.test.functional.ScanSessionTimeOutIT
   org.apache.accumulo.test.CreateTableWithNewTableConfigIT
   org.apache.accumulo.test.functional.SparseColumnFamilyIT
   org.apache.accumulo.test.functional.WatchTheWatchCountIT
   org.apache.accumulo.test.replication.StatusCombinerMacIT
   org.apache.accumulo.test.functional.SslWithClientAuthIT
   org.apache.accumulo.test.functional.SessionDurabilityIT
   org.apache.accumulo.test.functional.RegexGroupBalanceIT
   org.apache.accumulo.test.functional.MetadataMaxFilesIT
   org.apache.accumulo.test.functional.GarbageCollectorIT
   org.apache.accumulo.test.functional.BatchWriterFlushIT
   org.apache.accumulo.test.MasterRepairsDualAssignmentIT
   org.apache.accumulo.test.functional.ZookeeperRestartIT
   org.apache.accumulo.test.functional.MasterAssignmentIT
   org.apache.accumulo.test.functional.DeleteEverythingIT
   org.apache.accumulo.test.ConfigurableMajorCompactionIT
   org.apache.accumulo.test.functional.DeleteRowsSplitIT
   org.apache.accumulo.test.functional.LateLastContactIT
   org.apache.accumulo.test.functional.HalfDeadTServerIT
   org.apache.accumulo.test.functional.ChaoticBalancerIT
   org.apache.accumulo.test.functional.BadIteratorMincIT
   org.apache.accumulo.test.functional.ServerSideErrorIT
   org.apache.accumulo.test.functional.FateStarvationIT
   org.apache.accumulo.test.functional.MasterFailoverIT
   org.apache.accumulo.test.functional.BatchScanSplitIT
   org.apache.accumulo.test.functional.MonitorLoggingIT
   org.apache.accumulo.test.proxy.TJsonProtocolProxyIT
   org.apache.accumulo.test.RewriteTabletDirectoriesIT
   org.apache.accumulo.test.functional.RestartStressIT
   org.apache.accumulo.test.functional.BigRootTabletIT
   org.apache.accumulo.test.functional.KerberosProxyIT
   org.apache.accumulo.test.functional.WriteAheadLogIT
   org.apache.accumulo.test.functional.MetadataSplitIT
   org.apache.accumulo.test.TableConfigurationUpdateIT
   org.apache.accumulo.test.functional.SplitRecoveryIT
   org.apache.accumulo.test.ArbitraryTablePropertiesIT
   org.apache.accumulo.test.replication.ReplicationIT
   org.apache.accumulo.test.functional.CreateAndUseIT
   org.apache.accumulo.test.BadDeleteMarkersCreatedIT
   org.apache.accumulo.test.functional.BinaryStressIT
   org.apache.accumulo.test.functional.BackupMasterIT
   org.apache.accumulo.test.DetectDeadTabletServersIT
   org.apache.accumulo.test.functional.ScanIteratorIT
   org.apache.accumulo.test.BalanceWithOfflineTableIT
   org.apache.accumulo.test.TabletServerHdfsRestartIT
   org.apache.accumulo.test.functional.BloomFilterIT
   org.apache.accumulo.test.UserCompactionStrategyIT
   org.apache.accumulo.test.functional.WALSunnyDayIT
   org.apache.accumulo.test.functional.LogicalTimeIT
   org.apache.accumulo.test.functional.ClassLoaderIT
   org.apache.accumulo.test.functional.PermissionsIT
   org.apache.accumulo.test.functional.ConcurrencyIT
   org.apache.accumulo.test.functional.CredentialsIT
  

Re: Separate Performance Tests execution documentation

[Eric Newton]

I would suggest putting it in Javadoc in the PerformanceTest interface.

On Tue, Jul 7, 2015 at 12:24 PM, John Vines vi...@apache.org wrote:

 README in the top of the testing module?

 On Tue, Jul 7, 2015 at 12:18 PM Josh Elser josh.el...@gmail.com wrote:

  I committed https://issues.apache.org/jira/browse/ACCUMULO-3929
 yesterday.
 
  Had a thought today that it might be beneficial to write down how it
  works somewhere, but I don't know where would be best? Any suggestions?
 

Re: Post 1.5.3 and 1.6.3

[Eric Newton]

More importantly, when are we going to have a happy hour to celebrate?

-Eric


On Mon, Jul 6, 2015 at 4:04 PM, Josh Elser josh.el...@gmail.com wrote:

 Thanks to the efforts spearheaded by Christopher and verified by everyone
 else, we now have 1.5.3 and 1.6.3 releases!

 To keep the ball rolling, what's next? High level questions that come to
 mind...

 * When do we do 1.7.1 and/or 1.8.0?
 * What bug-fixes do we have outstanding for 1.7.1?
 * What other minor improvements do people want for 1.8.0?
 * Where does 2.0.0 stand? Should we make a bigger effort to getting the
 new client API stuff Christopher had started into Apache?

 Feel free to brainstorm here and/or on JIRA (tagging relevant issues to
 the desired fixVersion)

 - Josh

Re: Separation of timing/performance tests from normal Maven build

[Eric Newton]

+1 this is worth doing.  I don't know how, though. Yet.

On Wed, Jul 1, 2015 at 2:16 PM, Mike Drob md...@mdrob.com wrote:

 +1 annotate categories

 On Wed, Jul 1, 2015 at 1:09 PM, Josh Elser josh.el...@gmail.com wrote:

  Was talking with Eric off-list about a recent test he added.
 
  Over the past two major release lines (1.6 and 1.7), there's been a
  significant level of effort put forth by multiple devs to get the
  integration tests running on terrible hardware. This has been a great
  endeavor because our tests have never been more stable and it's even
 helped
  us catch bugs that we would have otherwise assumed as transiently failing
  (ACCUMULO-3859 is a great example).
 
  Because we are writing a database, we're always concerned about
  performance regressions, both high-level and low-level. I'd like to
 propose
  that we recognize and accept this head-on and try to move these
  specifically high-load and performance related tests to their own
  execution phase that we can run specifically on nodes that meet the
  necessary preconditions.
 
  Some examples of tests:
 
  DeleteTableDuringSplitIT
  DurabilityIT
  ManySplitIT
  RollWALPerformanceIT
 
  I know we can do some classification of tests via surefire/failsafe which
  should roughly meet our goals (typically via an annotation on the class).
  Thus, we could add a specific flag to a Maven build that would include
 this
  subset of tests.
 
  What do people think?  Do others also think that this is worth pursuing?
 
  - Josh
 

Re: Start scripts and address/hostname

[Eric Newton]

Our ops people use the start-here.sh scripts to bring services back up
after failures.  That's a great convenience: they don't have to remember
which hosts are supposed to run the which service.

In sympathy with your hostname troubles: the inconsistent use of hostname
determination causes those tservers started with start-all.sh and
start-here.sh to have different hostnames (shortname and fqdn,
respectively). This has something to do with how our DNS is set-up (or
hardcoded) because I cannot reproduce the effect in my development
environment.

As a consequence of this, the quoting hell of ssh, the limitations of
writing code in Bash, I'm avoiding The Scripts as much as possible.  I am
happy you are taking this on.

-Eric

On Thu, Jun 25, 2015 at 1:24 PM, Josh Elser josh.el...@gmail.com wrote:

 I've been on a tear within our scripts in the last day. I've been moving
 towards getting an accumulo-daemon.sh with some reasonable start, stop, etc
 semantics (ala Hadoop). This can also be done without affecting the
 existing start-server.sh, start-here.sh, etc scripts.

 This hypothetical accumulo-daemon.sh script is a close feel to what an
 init.d script would do. It alters the state of a server process on the
 local node. One thing I'm struggling to wrangle is the current ability the
 scripts/configs provide to control the interface that the server processes
 bind to.

 For example, 127.0.0.1 in the `slaves` file will result in a TabletServer
 that processes external to the local node cannot talk to. I know there are
 likely fringe cases (multiple NICs, bonded interface) which I don't fully
 understand to ensure proper support.

 Is anyone an expert here and could give some advice about the kinds of
 configuration that the scripts should provide to lets users run Accumulo
 how they want to? I would like to move away from having to pass the
 hostname/IP to scripts locally (e.g. `accumulo-daemon.sh start tserver`
 would start a tserver locally), but I don't want break an existing
 deployment.

 - Josh

Re: [VOTE] Accumulo 1.5.3-rc1

[Eric Newton]

+1

Verified:
* gpg signatures
* source matches tag
* user manual generated properly
* basic start-up and run on a single node against hadoop 2.6.0


On Sat, Jun 20, 2015 at 10:52 PM, Christopher ctubb...@apache.org wrote:

 This vote has failed due to an insufficient number of votes. As such,
 I'm extending it another 4 days from now (this is equivalent to
 calling a new vote with a duration of 4 days, using the same
 artifacts).

 This new vote will end on Thu Jun 25 03:00:00 UTC 2015
 (Wed Jun 24 23:00:00 EDT 2015 / Wed Jun 24 20:00:00 PDT 2015)

 --
 Christopher L Tubbs II
 http://gravatar.com/ctubbsii


 On Wed, Jun 17, 2015 at 7:12 PM, Christopher ctubb...@apache.org wrote:
  Accumulo Developers,
 
  Please consider the following candidate for Accumulo 1.5.3.
 
  Git Commit:
  9009d4897800a6a85e6a35297f9dd9880415a187
  Branch:
  1.5.3-rc1
 
  If this vote passes, a gpg-signed tag will be created using:
  git tag -f -m 'Apache Accumulo 1.5.3' -s 1.5.3
  9009d4897800a6a85e6a35297f9dd9880415a187
 
  Staging repo:
 https://repository.apache.org/content/repositories/orgapacheaccumulo-1036
  Source (official release artifact):
 
 https://repository.apache.org/content/repositories/orgapacheaccumulo-1036/org/apache/accumulo/accumulo/1.5.3/accumulo-1.5.3-src.tar.gz
  Binary:
 https://repository.apache.org/content/repositories/orgapacheaccumulo-1036/org/apache/accumulo/accumulo/1.5.3/accumulo-1.5.3-bin.tar.gz
  (Append .sha1, .md5, or .asc to download the signature/hash for
  a given artifact.)
 
  All artifacts were built and staged with:
  mvn release:prepare  mvn release:perform
 
  Signing keys are available at https://www.apache.org/dist/accumulo/KEYS
  (Expected fingerprint: 8CC4F8A2B29C2B040F2B835D6F0CDAE700B6899D)
 
  Release notes (in progress) can be found at
  https://accumulo.apache.org/release_notes/1.5.3
 
  Please vote one of:
  [ ] +1 - I have verified and accept...
  [ ] +0 - I have reservations, but not strong enough to vote against...
  [ ] -1 - Because..., I do not accept...
  ... these artifacts as the 1.5.3 release of Apache Accumulo.
 
  This vote will end on Sat Jun 20 23:30:00 UTC 2015
  (Sat Jun 20 19:30:00 EDT 2015 / Sat Jun 20 16:30:00 PDT 2015)
 
  Thanks!
 
  P.S. Hint: download the whole staging repo with
  wget -erobots=off -r -l inf -np -nH \
 
 https://repository.apache.org/content/repositories/orgapacheaccumulo-1036/
  # note the trailing slash is needed
 
 
  --
  Christopher L Tubbs II
  http://gravatar.com/ctubbsii

Re: [VOTE] Accumulo 1.5.3-rc1

[Eric Newton]

Also verified all unit and integration tests (mvn verify).

On Tue, Jun 23, 2015 at 12:02 PM, Eric Newton eric.new...@gmail.com wrote:

 +1

 Verified:
 * gpg signatures
 * source matches tag
 * user manual generated properly
 * basic start-up and run on a single node against hadoop 2.6.0


 On Sat, Jun 20, 2015 at 10:52 PM, Christopher ctubb...@apache.org wrote:

 This vote has failed due to an insufficient number of votes. As such,
 I'm extending it another 4 days from now (this is equivalent to
 calling a new vote with a duration of 4 days, using the same
 artifacts).

 This new vote will end on Thu Jun 25 03:00:00 UTC 2015
 (Wed Jun 24 23:00:00 EDT 2015 / Wed Jun 24 20:00:00 PDT 2015)

 --
 Christopher L Tubbs II
 http://gravatar.com/ctubbsii


 On Wed, Jun 17, 2015 at 7:12 PM, Christopher ctubb...@apache.org wrote:
  Accumulo Developers,
 
  Please consider the following candidate for Accumulo 1.5.3.
 
  Git Commit:
  9009d4897800a6a85e6a35297f9dd9880415a187
  Branch:
  1.5.3-rc1
 
  If this vote passes, a gpg-signed tag will be created using:
  git tag -f -m 'Apache Accumulo 1.5.3' -s 1.5.3
  9009d4897800a6a85e6a35297f9dd9880415a187
 
  Staging repo:
 https://repository.apache.org/content/repositories/orgapacheaccumulo-1036
  Source (official release artifact):
 
 https://repository.apache.org/content/repositories/orgapacheaccumulo-1036/org/apache/accumulo/accumulo/1.5.3/accumulo-1.5.3-src.tar.gz
  Binary:
 https://repository.apache.org/content/repositories/orgapacheaccumulo-1036/org/apache/accumulo/accumulo/1.5.3/accumulo-1.5.3-bin.tar.gz
  (Append .sha1, .md5, or .asc to download the signature/hash for
  a given artifact.)
 
  All artifacts were built and staged with:
  mvn release:prepare  mvn release:perform
 
  Signing keys are available at https://www.apache.org/dist/accumulo/KEYS
  (Expected fingerprint: 8CC4F8A2B29C2B040F2B835D6F0CDAE700B6899D)
 
  Release notes (in progress) can be found at
  https://accumulo.apache.org/release_notes/1.5.3
 
  Please vote one of:
  [ ] +1 - I have verified and accept...
  [ ] +0 - I have reservations, but not strong enough to vote against...
  [ ] -1 - Because..., I do not accept...
  ... these artifacts as the 1.5.3 release of Apache Accumulo.
 
  This vote will end on Sat Jun 20 23:30:00 UTC 2015
  (Sat Jun 20 19:30:00 EDT 2015 / Sat Jun 20 16:30:00 PDT 2015)
 
  Thanks!
 
  P.S. Hint: download the whole staging repo with
  wget -erobots=off -r -l inf -np -nH \
 
 https://repository.apache.org/content/repositories/orgapacheaccumulo-1036/
  # note the trailing slash is needed
 
 
  --
  Christopher L Tubbs II
  http://gravatar.com/ctubbsii

Re: Accumulo Carbon Daemon + Graphite

[Eric Newton]

I have created a simple port of OpenTSDB to accumulo [1].

I see someone else has created a pluggable storage mechanism for Graphite
[2].

-Eric

1. https://github.com/ericnewton/opentsdb
2. https://github.com/jbooth/graphite-data

On Mon, Jun 8, 2015 at 10:23 PM, Josh Elser josh.el...@gmail.com wrote:

 I can't say I have loads of free cycles to help, nor even know where to
 begin, but it appears that it would be a neat integration.

 Internally, we mull around every now and again about how better we can
 understand the system itself. Anything that might help us get this is good
 endeavor :)


 Michael Moss wrote:

 Hi, everyone.

 I've done a bit of googling and didn't see much, so I'm curious if anyone
 has anything to share about implementing a Graphite Carbon Daemon
 interface
 on top of Accumulo, similar to https://github.com/pyr/cyanite for
 Cassandra.

 It seems like everyone that uses Graphite eventually runs into scalability
 limitations with Whisper and there is a lot of interest in technologies
 like InfluxDB for metrics storage, but I feel like Accumulo has good
 ingredients for this use case itself!

 Interested in hearing what folks have to say and happy to lead some
 development for it as well.

 Thanks!

 -Mike

ACCUMULO-3871

[Eric Newton]

Hi Devs,

I've been working on running our long-running tests via map-reduce in order
to speed up testing.

In particular, when making sweeping changes to critical code (say,
ACCUMULO-3423 (1) ), it's helpful to re-run these tests quickly to make
sure that instability hasn't been introduced.  The 3.5 hour turn-around
time is not helpful for release testing, either.

I made a comment (2), and would like some feedback on the proposed
restructuring.

Basically, it would break the IT's out into their own project.

-Eric

1. https://issues.apache.org/jira/browse/ACCUMULO-3423
2.
https://issues.apache.org/jira/browse/ACCUMULO-3871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14572979#comment-14572979

Re: Configure accumulo-env.sh with 40GB of memory

[Eric Newton]

The apache mailing lists do not pass through email attachments.  You could
try posting your config on https://paste.apache.org/.

150K / node for small, independent mutations is very good.

Make sure you have 20-80 splits (total) per node, for the tables you are
ingesting.  You may be limited by the number / threads you have in your
client.

-Eric

On Thu, Jun 4, 2015 at 4:58 AM, Revan1988 andrealeon...@gmail.com wrote:

 Hi,
 In my server (only one cluster) i've got 40GB of memory size.
 I configured my accumulo-env.sh in this way:



 is it a good configuration?
 on my hardware (8 core/40gb/btfs hdd 7200) i've got about 50k~150k
 ingest/second... it's a good performance?

 
 http://apache-accumulo.1065345.n5.nabble.com/file/n14304/Schermata_2015-06-04_alle_10.png
 


 Thank you very much!



 -
 Andrea Leoni
 Italy
 Computer Engineer
 --
 View this message in context:
 http://apache-accumulo.1065345.n5.nabble.com/Configure-accumulo-env-sh-with-40GB-of-memory-tp14304.html
 Sent from the Developers mailing list archive at Nabble.com.

Re: [DISCUSS] EOL 1.5

[Eric Newton]

+1 for EOL for 1.5.

Making fixes in 1.5 and then merging it to 1.6, 1.7, and then master is
tedious work. 1.5 makes the task more challenging because the layout of the
packages changed so much in 1.6.

-Eric


On Thu, May 21, 2015 at 1:18 PM, Christopher ctubb...@apache.org wrote:

 So, at this point, I'm willing to do a 1.5.3 release and can start
 that today. It seems we're in agreement we should at least do that.
 Beyond that, I'm not really sure what the consensus is.

 --
 Christopher L Tubbs II
 http://gravatar.com/ctubbsii


 On Tue, May 12, 2015 at 10:35 PM, Josh Elser josh.el...@gmail.com wrote:
  1.5 has already started to suffer in terms of landing every bug-fix
 there. I
  don't think it's intentional (I know I have done it though), but it's
 kind
  of a sign that the devs have already mentally move beyond 1.5.
 
  I think JIRA is a clear sign that users aren't heavily using 1.5 (I can't
  think of more than a couple tickets marked as affects 1.5.x), but it
 would
  be nice to explicitly ask user@.
 
  A 1.5.3 to close things out would be nice -- can always be re-opened if
  someone wants to scratch that itch.
 
 
  Sean Busbey wrote:
 
  that change to development procedure will definitely impact them. it'll
  mean folks no longer look for their bugs to impact the 1.5 branch to
 start
  (unless things are critical). that basically guarantees that the rate of
  1.5 releases will slow, which impacts ops planning for those on the 1.5
  line.
 
  On Tue, May 12, 2015 at 2:42 PM, Christopherctubb...@apache.org
 wrote:
 
  Feel free to include user@ sooner, if you wish. The reason I hadn't
  already was because my suggested route would only be a shift in
  development procedures, and wouldn't really change things from a user
  perspective. Alternatives to what I suggest may affect them more
  strongly. We definitely should CC them when we have a decision,
  though.
 
  --
  Christopher L Tubbs II
  http://gravatar.com/ctubbsii
 
 
  On Tue, May 12, 2015 at 2:55 PM, Sean Busbeybus...@cloudera.com
 wrote:
 
  oh! almost forgot. We should get user@accumulo into this conversation
  sooner rather than later. I'm not sure if it's  better ot just copy
 them
 
  in
 
  to this thread or do it as a follow up once we have more of an idea of
 
  what
 
  EOL means for them.
 
  On Tue, May 12, 2015 at 1:53 PM, Sean Busbeybus...@cloudera.com
 
  wrote:
 
  +1 to making sure we have a 1.5.3 before stop dev
 
  I'd like to make sure we get through some testing of 1.5 -  1.7
  upgrade
  testing before declaring dev over, just to give people more assurance
 
  that
 
  they can upgrade off of the version.
 
  On Tue, May 12, 2015 at 1:18 PM, Christopherctubb...@apache.org
 
  wrote:
 
  How do we want to EOL 1.5?
 
  Personally, I was thinking (soon after 1.7.0 is released):
  * Release and tag 1.5.3
  * Remove 1.5 branch to focus active development on newer versions
  * Be willing to branch from the 1.5.3 tag to rapidly release a 1.5.4
  in response to critical bugs
 
  My biggest concerns are:
  1) We turn exhausted people off by doing burdensome release testing,
  which delays bugfixes in 1.5, and
  2) We get into a situation where 1.5.3 has some bugs that we never
  fix, which sends a confusing message to stick with 1.5.2.
 
  There's also the concern that there's a fair amount of work that was
  put into 1.5.3, and I'd hate to have those contributions not be
  available to users of 1.5.
 
  I figure that so long as we're willing to fix critical bugs, we can
  formally cease active development (EOL), without going so far as to
  say that 1.5 users are completely screwed if a critical bug is
  identified.
 
  What I'm describing isn't really an EOL date, so much as an EOL
 period
  which begins when we cease active development on 1.5, and ends
  organically at some arbitrary point in the future when people stop
  reporting critical bugs (or we reach a point where maintaining it is
  too costly... a sort of EOL-2).
 
  Another way to look at what I'm suggesting is switch from a
 sustained
  development model to a branch to fix and release model, where
  patch/bugfix releases are more narrowly scoped and can occur more
  rapidly, on demand.
 
  Thoughts? Alternatives? Variations? Objections?
 
  --
  Christopher L Tubbs II
  http://gravatar.com/ctubbsii
 
 
 
  --
  Sean
 
 
 
  --
  Sean
 
 
 
 
 

Re: [VOTE] Apache Accumulo 1.7.0-rc3

[Eric Newton]

I've been running a RW test w/out agitation on a 21 node EC2 cluster.

It died in the replication test.

So I restarted it with one walker, just doing replication.

Half the cluster died with zookeeper timeouts. I am investigating.

I'll pull the source, build and review the results by EOB today.

-Eric


On Thu, May 14, 2015 at 1:38 PM, Josh Elser josh.el...@gmail.com wrote:

 Reminder, ~27hrs remaining on this vote. 5pm ET tmrw.

 I know some people are running tests. Please pull down the source tarball,
 run unit tests, build the code, poke at it locally. Any and all feedback is
 warmly welcomed. The more eyes we have looking at this the better. Thanks
 in advance.


 Josh Elser wrote:

 You are correct. I forgot to update the SHA when I copied the contents.

 The correct SHA1=8cba8128fbc3238bdd9398cf5c36b7cb6dc3b61d

 Christopher wrote:

 The SHA1 seems incorrect. The jars says they were built on
 8cba8128fbc3238bdd9398cf5c36b7cb6dc3b61d, which I can't find in the RC
 branch.

 --
 Christopher L Tubbs II
 http://gravatar.com/ctubbsii


 On Tue, May 12, 2015 at 3:08 PM, Josh Elserjosh.el...@gmail.com wrote:

 Devs,

 Please consider the following candidate for Apache Accumulo 1.7.0

 Tag: 1.7.0-rc3
 SHA1: 76634fb2f1257abbb8ef745ea67a4f78e733a402
 Staging Repository:

 https://repository.apache.org/content/repositories/orgapacheaccumulo-1032


 Source tarball:

 https://repository.apache.org/content/repositories/orgapacheaccumulo-1032/org/apache/accumulo/accumulo/1.7.0/accumulo-1.7.0-src.tar.gz

 Binary tarball:

 https://repository.apache.org/content/repositories/orgapacheaccumulo-1032/org/apache/accumulo/accumulo/1.7.0/accumulo-1.7.0-bin.tar.gz

 (Append .sha1, .md5 or .asc to download the signature/hash for an
 artifact.)

 Signing keys available at: https://www.apache.org/dist/accumulo/KEYS

 1.7.0 includes 693 resolved issues:

 https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12312121version=12324607


 Testing: All unit and integration tests are passing. Completed 3-day
 CI w/o
 agitation or verification. Completed 24hr RandomWalk w/o agitation on 3
 nodes. 95% through 24hr CI and RW w/ agitation.

 API compatibility report for 1.6.2 to 1.7.0:

 http://people.apache.org/~elserj/accumulo-1.7.0-rc3/1.6.2_to_1.7.0/compat_report.html


 API backwards compatibility report for 1.7.0 to 1.6.2:

 http://people.apache.org/~elserj/accumulo-1.7.0-rc3/1.7.0_to_1.6.2/compat_report.html


 The vote will be open for 72hrs until Friday, May 16th 4:00PM ET.
 Here's my
 +1.

Re: New committer: Vikram

[Eric Newton]

Welcome, Vikram!


On Fri, May 15, 2015 at 5:26 PM, Keith Turner ke...@deenlo.com wrote:

 Welcome!

 On Fri, May 15, 2015 at 5:08 PM, Sean Busbey bus...@cloudera.com wrote:

  Hi Accumulo!
 
  The Apache Accumulo PMC recently voted in Vikram Srivastava as a
 committer
  and PMC in cognition of his past contributions to the project.
 
  Please give him a good welcome. Vikram, feel free to give everyone a
 brief
  introduction on yourself and your interests.
 
 
  --
  Sean
 

Re: [VOTE] Apache Accumulo 1.7.0-rc3

[Eric Newton]

+1

My RW testing went well enough, and I didn't see failures until the cluster
hung up.ACCUMULO-3818 is a test issue, and not a problem with accumulo.

I was able to build 1.7.0 from the source tarball.

I'll continue to investigate the recoverLease issue.

-Eric

On Fri, May 15, 2015 at 4:09 PM, Josh Elser josh.el...@gmail.com wrote:

 In some of my testing on HDFS 2.6.0 and agitation, I believe leases
 weren't being recovered until the datanode was restarted (which agitation
 killed). Every time I jstack'ed the master, I saw it hung on the HDFS
 recoverLease() method. I saw nothing that suggested a bug in Accumulo (a
 nod towards Christopher's assessment).


 Eric Newton wrote:

 Can you quantify Some lease recoveries taking a long time in HDFS.?

 I think Keith and I are ok with the release. We're looking into our
 (shared) testing environment to figure out why Accumulo and HDFS are not
 getting along during recoveries.

 -Eric


 On Fri, May 15, 2015 at 3:51 PM, Josh Elserjosh.el...@gmail.com  wrote:

  72hr 3node CI verify w/ agitation just succeeded.

 org.apache.accumulo.test.continuous.ContinuousVerify$Counts
  REFERENCED=19706560041
  UNREFERENCED=29844289

 Repeatedly ran into issues with HoldTime from HDFS (wrote a loop to just
 restart the ingester). Some lease recoveries taking a long time in HDFS.
 Nothing super impactful.

 I believe that wraps up all required testing.


 Josh Elser wrote:

  24hr CI just verified

 org.apache.accumulo.test.continuous.ContinuousVerify$Counts
 REFERENCED=5754999773
 UNREFERENCED=6000227

 Josh Elser wrote:

  24hr RW on 3 nodes w/ agitation just finished positively.

 Josh Elser wrote:

  Testing: All unit and integration tests are passing. Completed 3-day
 CI
 w/o agitation or verification. Completed 24hr RandomWalk w/o agitation
 on 3 nodes. 95% through 24hr CI and RW w/ agitation.

Re: [VOTE] Apache Accumulo 1.7.0-rc3

[Eric Newton]

Can you quantify Some lease recoveries taking a long time in HDFS.?

I think Keith and I are ok with the release. We're looking into our
(shared) testing environment to figure out why Accumulo and HDFS are not
getting along during recoveries.

-Eric


On Fri, May 15, 2015 at 3:51 PM, Josh Elser josh.el...@gmail.com wrote:

 72hr 3node CI verify w/ agitation just succeeded.

 org.apache.accumulo.test.continuous.ContinuousVerify$Counts
 REFERENCED=19706560041
 UNREFERENCED=29844289

 Repeatedly ran into issues with HoldTime from HDFS (wrote a loop to just
 restart the ingester). Some lease recoveries taking a long time in HDFS.
 Nothing super impactful.

 I believe that wraps up all required testing.


 Josh Elser wrote:

 24hr CI just verified

 org.apache.accumulo.test.continuous.ContinuousVerify$Counts
 REFERENCED=5754999773
 UNREFERENCED=6000227

 Josh Elser wrote:

 24hr RW on 3 nodes w/ agitation just finished positively.

 Josh Elser wrote:


 Testing: All unit and integration tests are passing. Completed 3-day CI
 w/o agitation or verification. Completed 24hr RandomWalk w/o agitation
 on 3 nodes. 95% through 24hr CI and RW w/ agitation.

Re: [VOTE] Apache Accumulo 1.7.0-rc3

[Eric Newton]

Even though I had EC2 agitating my tablet servers for me, the reason the
test failed was a failed recovery due to lease recovery failure.  This is
the same thing Keith saw with his CI test. We did make small changes to the
lease recovery code (dropped Hadoop 1 support), but I don't know why that
would make a difference.

The fact that we're seeing this problem in multiple tests is troubling.

I'm continuing to investigate the issue.

-Eric

On Fri, May 15, 2015 at 12:09 PM, Eric Newton eric.new...@gmail.com wrote:

 I've been running a RW test w/out agitation on a 21 node EC2 cluster.

 It died in the replication test.

 So I restarted it with one walker, just doing replication.

 Half the cluster died with zookeeper timeouts. I am investigating.

 I'll pull the source, build and review the results by EOB today.

 -Eric


 On Thu, May 14, 2015 at 1:38 PM, Josh Elser josh.el...@gmail.com wrote:

 Reminder, ~27hrs remaining on this vote. 5pm ET tmrw.

 I know some people are running tests. Please pull down the source
 tarball, run unit tests, build the code, poke at it locally. Any and all
 feedback is warmly welcomed. The more eyes we have looking at this the
 better. Thanks in advance.


 Josh Elser wrote:

 You are correct. I forgot to update the SHA when I copied the contents.

 The correct SHA1=8cba8128fbc3238bdd9398cf5c36b7cb6dc3b61d

 Christopher wrote:

 The SHA1 seems incorrect. The jars says they were built on
 8cba8128fbc3238bdd9398cf5c36b7cb6dc3b61d, which I can't find in the RC
 branch.

 --
 Christopher L Tubbs II
 http://gravatar.com/ctubbsii


 On Tue, May 12, 2015 at 3:08 PM, Josh Elserjosh.el...@gmail.com
 wrote:

 Devs,

 Please consider the following candidate for Apache Accumulo 1.7.0

 Tag: 1.7.0-rc3
 SHA1: 76634fb2f1257abbb8ef745ea67a4f78e733a402
 Staging Repository:

 https://repository.apache.org/content/repositories/orgapacheaccumulo-1032


 Source tarball:

 https://repository.apache.org/content/repositories/orgapacheaccumulo-1032/org/apache/accumulo/accumulo/1.7.0/accumulo-1.7.0-src.tar.gz

 Binary tarball:

 https://repository.apache.org/content/repositories/orgapacheaccumulo-1032/org/apache/accumulo/accumulo/1.7.0/accumulo-1.7.0-bin.tar.gz

 (Append .sha1, .md5 or .asc to download the signature/hash for an
 artifact.)

 Signing keys available at: https://www.apache.org/dist/accumulo/KEYS

 1.7.0 includes 693 resolved issues:

 https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12312121version=12324607


 Testing: All unit and integration tests are passing. Completed 3-day
 CI w/o
 agitation or verification. Completed 24hr RandomWalk w/o agitation on 3
 nodes. 95% through 24hr CI and RW w/ agitation.

 API compatibility report for 1.6.2 to 1.7.0:

 http://people.apache.org/~elserj/accumulo-1.7.0-rc3/1.6.2_to_1.7.0/compat_report.html


 API backwards compatibility report for 1.7.0 to 1.6.2:

 http://people.apache.org/~elserj/accumulo-1.7.0-rc3/1.7.0_to_1.6.2/compat_report.html


 The vote will be open for 72hrs until Friday, May 16th 4:00PM ET.
 Here's my
 +1.

Re: Launching Accumulo Servers Programatically

[Eric Newton]

An in-process accumulo would be a good replacement for Mock, which is dying
on the vine.

-Eric


On Mon, May 11, 2015 at 6:04 PM, dlmarion dlmar...@comcast.net wrote:



 Sorry for the short response, sitting at the ball field, but want to chime
 in. I have some interest in this area as well. I have some local code
 modifications that I intend to commit at some point to start / stop
 Accumulo. This involves making the admin methods programatically accessible.




  Original message 
 From: Jim Klucar klu...@gmail.com
 Date: 05/11/2015  5:09 PM  (GMT-05:00)
 To: dev@accumulo.apache.org
 Subject: Launching Accumulo Servers Programatically

 Devs,

 I've been working on the mesos-accumulo framework. For those not familiar
 with Mesos, it is a cluster resource manager that allows you to launch
 tasks across the cluster. To do that, you write a mesos framework.

 We have proven this is possible with accumulo by writing a quick Scala
 framework implementation that uses the accumulo script to launch servers
 from a command line. This worked fine but we would like more insight into
 the servers once they're launched, outside of does the pid still exist on
 the machine. What I would like to do is something like:

 org.apache.accumulo.server.Master master = new o.a.a.server.Master();

 from inside the mesos executor code. Then setup some config, or pass it in
 the constructor and call master.run() or whatever. The point is at the end
 of some procedure the server is running, and I can call methods like
 master.shutdown() or master.ping() or whatever else is exposed.  A nice
 standard server interface across server types would be great.

 I realize this may be quite a refactoring of all the server classes so I
 thought I'd gather some opinions on the idea here instead of just writing
 JIRA tickets.

 Jim

Re: Review Request 32224: ACCUMULO-3423

[Eric Newton]

 On March 20, 2015, 7:11 p.m., kturner wrote:
  This change make the case where a tablet has some non-empty WALs with no 
  data more likely (not sure how likely it would have been before).  For 
  example a tablet could have WAL1 with data for it, WAL2 with no data for 
  it, and WAL3 with data.   Seems like continuous ingest would not trigger 
  this case.  Wonder if an IT for this case is worthwhile.
  
  I am still reviewing.  Was getting nervous about RB losing my comments, so 
  posting some comments I made so far.

Added an IT for this case, in particular. Also added tests to ensure that empty 
logs for dead tablet servers are marked unused and are deleted.


 On March 20, 2015, 7:11 p.m., kturner wrote:
  server/base/src/main/java/org/apache/accumulo/server/master/state/MetaDataStateStore.java,
   line 139
  https://reviews.apache.org/r/32224/diff/1/?file=899480#file899480line139
 
  Seems like we could end with logs for multiple tservers in the 
  following case.
  
1. Tablet T1 is assigned to Tserver1
2. Tserver1.I1 dies
3. Master adds logs for Tserver1 to Tablet T1
4. Master assigns Tablet T1 to Tserver2
5. Before Tserver T2 loads and recovers Tablet T1, it dies
6. Master adds logs for Tserver2 to Tablet T1
  
  In this case Tablet T1 has walog from two tservers.  Not sure how well 
  recovery code would handle this.   Its seems like recovery code makes the 
  assumption that walogs are from one tserver.

Modified the code so that logs are only written if the tablet was actually 
hosted.  This should prevent logs from being added before the old logs are 
recovered.


 On March 20, 2015, 7:11 p.m., kturner wrote:
  server/master/src/main/java/org/apache/accumulo/master/TabletGroupWatcher.java,
   line 293
  https://reviews.apache.org/r/32224/diff/1/?file=899498#file899498line293
 
  If no tablets are assigned to a dead tserver, will its logs be marked 
  as unused?  Was wondering if this operation is idempotent.  If the master 
  dies right after flushChanges, then it seems like no tablets would be 
  assigned to the dead server.

Added a scan to the GC to remove logs without metadata reference for dead 
servers.


- Eric


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32224/#review77239
---


On March 19, 2015, 4:42 p.m., Eric Newton wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/32224/
 ---
 
 (Updated March 19, 2015, 4:42 p.m.)
 
 
 Review request for accumulo.
 
 
 Bugs: ACCUMULO-3423
 https://issues.apache.org/jira/browse/ACCUMULO-3423
 
 
 Repository: accumulo
 
 
 Description
 ---
 
 Faster WAL rollovers
 
 
 Diffs
 -
 
   
 core/src/main/java/org/apache/accumulo/core/client/impl/ReplicationOperationsImpl.java
  6a5c74a 
   core/src/main/java/org/apache/accumulo/core/conf/Property.java 2403915 
   core/src/main/java/org/apache/accumulo/core/metadata/RootTable.java 24148b1 
   
 core/src/main/java/org/apache/accumulo/core/metadata/schema/MetadataSchema.java
  534dd7f 
   core/src/main/java/org/apache/accumulo/core/tabletserver/log/LogEntry.java 
 25d0f32 
   
 core/src/test/java/org/apache/accumulo/core/metadata/MetadataTableSchemaTest.java
  PRE-CREATION 
   server/base/src/main/java/org/apache/accumulo/server/fs/VolumeUtil.java 
 fc26ca4 
   server/base/src/main/java/org/apache/accumulo/server/init/Initialize.java 
 e5bdded 
   
 server/base/src/main/java/org/apache/accumulo/server/master/state/MetaDataStateStore.java
  7ee6f0c 
   
 server/base/src/main/java/org/apache/accumulo/server/master/state/MetaDataTableScanner.java
  9be5a67 
   
 server/base/src/main/java/org/apache/accumulo/server/master/state/TabletLocationState.java
  b24b562 
   
 server/base/src/main/java/org/apache/accumulo/server/master/state/TabletStateStore.java
  5413e31 
   
 server/base/src/main/java/org/apache/accumulo/server/master/state/ZooTabletStateStore.java
  ab99396 
   
 server/base/src/main/java/org/apache/accumulo/server/replication/StatusUtil.java
  898e3d4 
   
 server/base/src/main/java/org/apache/accumulo/server/util/ListVolumesUsed.java
  e90d1dd 
   
 server/base/src/main/java/org/apache/accumulo/server/util/MasterMetadataUtil.java
  4ca2d64 
   
 server/base/src/main/java/org/apache/accumulo/server/util/MetadataTableUtil.java
  10cd749 
   
 server/base/src/main/java/org/apache/accumulo/server/util/ReplicationTableUtil.java
  af02a8d 
   
 server/base/src/test/java/org/apache/accumulo/server/util/ReplicationTableUtilTest.java
  b1010c2 
   
 server/gc/src/main/java/org/apache/accumulo/gc/GarbageCollectWriteAheadLogs.java
  1735c0d 
   server/gc/src/main/java/org/apache/accumulo/gc

Re: Review Request 32224: ACCUMULO-3423

[Eric Newton]

 On March 19, 2015, 8:45 p.m., Josh Elser wrote:
  core/src/main/java/org/apache/accumulo/core/tabletserver/log/LogEntry.java, 
  line 36
  https://reviews.apache.org/r/32224/diff/1/?file=899476#file899476line36
 
  As a follow-on, this would be a good class to get some explicit unit 
  test coverage for (since it acts less like an internal struct with your 
  changes)

Created a unit test for LogEntry


 On March 19, 2015, 8:45 p.m., Josh Elser wrote:
  core/src/main/java/org/apache/accumulo/core/tabletserver/log/LogEntry.java, 
  line 109
  https://reviews.apache.org/r/32224/diff/1/?file=899476#file899476line109
 
  Have you tested out a dirty 1.6 shutdown and starting up 1.7 with these 
  changes in place?

Yes, I've started this up on a dirty 1.6-SNAPSHOT


 On March 19, 2015, 8:45 p.m., Josh Elser wrote:
  server/tserver/src/main/java/org/apache/accumulo/tserver/log/DfsLogger.java,
   line 77
  https://reviews.apache.org/r/32224/diff/1/?file=899508#file899508line77
 
  Why is DfsLogger Comparable now? Isn't the log name still just a UUID?

DfsLogger is now stored in a ConcurrentSkipList which needs it to be Comparable.


- Eric


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32224/#review77098
---


On March 19, 2015, 4:42 p.m., Eric Newton wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/32224/
 ---
 
 (Updated March 19, 2015, 4:42 p.m.)
 
 
 Review request for accumulo.
 
 
 Bugs: ACCUMULO-3423
 https://issues.apache.org/jira/browse/ACCUMULO-3423
 
 
 Repository: accumulo
 
 
 Description
 ---
 
 Faster WAL rollovers
 
 
 Diffs
 -
 
   
 core/src/main/java/org/apache/accumulo/core/client/impl/ReplicationOperationsImpl.java
  6a5c74a 
   core/src/main/java/org/apache/accumulo/core/conf/Property.java 2403915 
   core/src/main/java/org/apache/accumulo/core/metadata/RootTable.java 24148b1 
   
 core/src/main/java/org/apache/accumulo/core/metadata/schema/MetadataSchema.java
  534dd7f 
   core/src/main/java/org/apache/accumulo/core/tabletserver/log/LogEntry.java 
 25d0f32 
   
 core/src/test/java/org/apache/accumulo/core/metadata/MetadataTableSchemaTest.java
  PRE-CREATION 
   server/base/src/main/java/org/apache/accumulo/server/fs/VolumeUtil.java 
 fc26ca4 
   server/base/src/main/java/org/apache/accumulo/server/init/Initialize.java 
 e5bdded 
   
 server/base/src/main/java/org/apache/accumulo/server/master/state/MetaDataStateStore.java
  7ee6f0c 
   
 server/base/src/main/java/org/apache/accumulo/server/master/state/MetaDataTableScanner.java
  9be5a67 
   
 server/base/src/main/java/org/apache/accumulo/server/master/state/TabletLocationState.java
  b24b562 
   
 server/base/src/main/java/org/apache/accumulo/server/master/state/TabletStateStore.java
  5413e31 
   
 server/base/src/main/java/org/apache/accumulo/server/master/state/ZooTabletStateStore.java
  ab99396 
   
 server/base/src/main/java/org/apache/accumulo/server/replication/StatusUtil.java
  898e3d4 
   
 server/base/src/main/java/org/apache/accumulo/server/util/ListVolumesUsed.java
  e90d1dd 
   
 server/base/src/main/java/org/apache/accumulo/server/util/MasterMetadataUtil.java
  4ca2d64 
   
 server/base/src/main/java/org/apache/accumulo/server/util/MetadataTableUtil.java
  10cd749 
   
 server/base/src/main/java/org/apache/accumulo/server/util/ReplicationTableUtil.java
  af02a8d 
   
 server/base/src/test/java/org/apache/accumulo/server/util/ReplicationTableUtilTest.java
  b1010c2 
   
 server/gc/src/main/java/org/apache/accumulo/gc/GarbageCollectWriteAheadLogs.java
  1735c0d 
   server/gc/src/main/java/org/apache/accumulo/gc/SimpleGarbageCollector.java 
 c8d5cd6 
   
 server/gc/src/main/java/org/apache/accumulo/gc/replication/CloseWriteAheadLogReferences.java
  3a32727 
   
 server/gc/src/test/java/org/apache/accumulo/gc/GarbageCollectWriteAheadLogsTest.java
  5801faa 
   
 server/gc/src/test/java/org/apache/accumulo/gc/replication/CloseWriteAheadLogReferencesTest.java
  23db83a 
   server/master/src/main/java/org/apache/accumulo/master/Master.java 3762f32 
   
 server/master/src/main/java/org/apache/accumulo/master/MasterClientServiceHandler.java
  f73c236 
   
 server/master/src/main/java/org/apache/accumulo/master/TabletGroupWatcher.java
  d097d75 
   
 server/master/src/main/java/org/apache/accumulo/master/replication/WorkMaker.java
  8532e1b 
   
 server/master/src/main/java/org/apache/accumulo/master/state/MergeStats.java 
 44f229e 
   
 server/master/src/test/java/org/apache/accumulo/master/ReplicationOperationsImplTest.java
  a127dcd 
   server/master/src/test/java/org/apache/accumulo/master/TestMergeState.java 
 b0240f1 
   
 server/master/src/test/java/org/apache/accumulo/master

Re: Review Request 32224: ACCUMULO-3423

[Eric Newton]

 On March 27, 2015, 3:43 p.m., kturner wrote:
  core/src/main/java/org/apache/accumulo/core/tabletserver/log/LogEntry.java, 
  line 125
  https://reviews.apache.org/r/32224/diff/1/?file=899476#file899476line125
 
  should there be a sanity check on the length of `parts[]`?  Maybe the 
  construction should do the sanity check

Added a unit test for LogEntry, and check the format of the string a little 
more carefully.


 On March 27, 2015, 3:43 p.m., kturner wrote:
  server/tserver/src/main/java/org/apache/accumulo/tserver/TabletServer.java, 
  line 1783
  https://reviews.apache.org/r/32224/diff/1/?file=899507#file899507line1783
 
  why not just remove this thrift call?

[~elserj] wanted it in there.


 On March 27, 2015, 3:43 p.m., kturner wrote:
  server/master/src/main/java/org/apache/accumulo/master/TabletGroupWatcher.java,
   line 292
  https://reviews.apache.org/r/32224/diff/1/?file=899498#file899498line292
 
  The code to merge tablets had checks to ensure it would not merge 
  tablets with walogs.  Need to ensure the updates to add walogs to a tablet 
  are always made before those checks.   Looking at the code, it seems like 
  this may be the case but I am not completely sure.   Maybe some test for 
  this situation?  Maybe shard and bulk random walk with agitation is enought 
  to test this situation.  Do you think ITs are needed?

The only way to get unassigned by the master is to get the list of WALogs added 
at the same time.


 On March 27, 2015, 3:43 p.m., kturner wrote:
  server/tserver/src/main/java/org/apache/accumulo/tserver/log/TabletServerLogger.java,
   line 210
  https://reviews.apache.org/r/32224/diff/1/?file=899510#file899510line210
 
  I think the following code to create a log could be removed with the 
  SynchronousQueue... just always take from the syncronous queue

Good catch.  That improved the code a great deal.


 On March 27, 2015, 3:43 p.m., kturner wrote:
  server/tserver/src/main/java/org/apache/accumulo/tserver/log/TabletServerLogger.java,
   line 243
  https://reviews.apache.org/r/32224/diff/1/?file=899510#file899510line243
 
  why not just move check up to beginning and bail if there is already a 
  next log?

Used the SynchronousQueue as suggested.


 On March 27, 2015, 3:43 p.m., kturner wrote:
  server/tserver/src/main/java/org/apache/accumulo/tserver/log/TabletServerLogger.java,
   line 240
  https://reviews.apache.org/r/32224/diff/1/?file=899510#file899510line240
 
  Could iterate through tablets and find levels present, then pass those 
  levels to `addLoggersToMetadata()`.  This would avoid synchronizing on each 
  tablet.

Good idea.


 On March 27, 2015, 3:43 p.m., kturner wrote:
  server/tserver/src/main/java/org/apache/accumulo/tserver/log/TabletServerLogger.java,
   line 245
  https://reviews.apache.org/r/32224/diff/1/?file=899510#file899510line245
 
  Is this expected to occur often?  Should this unused log be marked 
  unused in  metadata table?

Code has been restructured using the SynchronousQueue, and this warning will 
never occure.


- Eric


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32224/#review78049
---


On March 19, 2015, 4:42 p.m., Eric Newton wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/32224/
 ---
 
 (Updated March 19, 2015, 4:42 p.m.)
 
 
 Review request for accumulo.
 
 
 Bugs: ACCUMULO-3423
 https://issues.apache.org/jira/browse/ACCUMULO-3423
 
 
 Repository: accumulo
 
 
 Description
 ---
 
 Faster WAL rollovers
 
 
 Diffs
 -
 
   
 core/src/main/java/org/apache/accumulo/core/client/impl/ReplicationOperationsImpl.java
  6a5c74a 
   core/src/main/java/org/apache/accumulo/core/conf/Property.java 2403915 
   core/src/main/java/org/apache/accumulo/core/metadata/RootTable.java 24148b1 
   
 core/src/main/java/org/apache/accumulo/core/metadata/schema/MetadataSchema.java
  534dd7f 
   core/src/main/java/org/apache/accumulo/core/tabletserver/log/LogEntry.java 
 25d0f32 
   
 core/src/test/java/org/apache/accumulo/core/metadata/MetadataTableSchemaTest.java
  PRE-CREATION 
   server/base/src/main/java/org/apache/accumulo/server/fs/VolumeUtil.java 
 fc26ca4 
   server/base/src/main/java/org/apache/accumulo/server/init/Initialize.java 
 e5bdded 
   
 server/base/src/main/java/org/apache/accumulo/server/master/state/MetaDataStateStore.java
  7ee6f0c 
   
 server/base/src/main/java/org/apache/accumulo/server/master/state/MetaDataTableScanner.java
  9be5a67 
   
 server/base/src/main/java/org/apache/accumulo/server/master/state/TabletLocationState.java
  b24b562 
   
 server/base/src/main/java/org/apache/accumulo/server/master/state/TabletStateStore.java
  5413e31 
   
 server

Re: Review Request 32224: ACCUMULO-3423

[Eric Newton]

 On March 23, 2015, 5:39 p.m., kturner wrote:
  server/base/src/main/java/org/apache/accumulo/server/util/MetadataTableUtil.java,
   line 1154
  https://reviews.apache.org/r/32224/diff/1/?file=899488#file899488line1154
 
  Why does this write to root and meta?

There could be a marker at the meta and/or root level, mark them both as 
unused. Comment added.


 On March 23, 2015, 5:39 p.m., kturner wrote:
  server/base/src/main/java/org/apache/accumulo/server/util/MetadataTableUtil.java,
   line 1199
  https://reviews.apache.org/r/32224/diff/1/?file=899488#file899488line1199
 
  What prevents unused logs from being fetched?

Good catch. I've fixed the code.


 On March 23, 2015, 5:39 p.m., kturner wrote:
  server/master/src/main/java/org/apache/accumulo/master/TabletGroupWatcher.java,
   line 232
  https://reviews.apache.org/r/32224/diff/1/?file=899498#file899498line232
 
  Is this code still needed?  Would it still be used for upgrade case?

I tried to make ACCUMULO-3423 as conservative as possible (yes, really!). To 
that end, log entries are put onto tablets for recovery, so all the trickiness 
of recovery could stay the same.


 On March 23, 2015, 5:39 p.m., kturner wrote:
  server/master/src/main/java/org/apache/accumulo/master/TabletGroupWatcher.java,
   line 273
  https://reviews.apache.org/r/32224/diff/1/?file=899498#file899498line273
 
  Would you expect this method to be called if the logs had already been 
  marked as unused?  It seems like it should not, but I am not 100% sure.  
  Thinking that if this is an unexpected case that a sanity check might be 
  worthwhile.

The metadata update to unassign a tablet creates the log references in the same 
mutation.  It should never be the case that the tablet is ASSIGNED to dead 
server and not need to fetch the logs.


- Eric


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32224/#review77413
---


On March 19, 2015, 4:42 p.m., Eric Newton wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/32224/
 ---
 
 (Updated March 19, 2015, 4:42 p.m.)
 
 
 Review request for accumulo.
 
 
 Bugs: ACCUMULO-3423
 https://issues.apache.org/jira/browse/ACCUMULO-3423
 
 
 Repository: accumulo
 
 
 Description
 ---
 
 Faster WAL rollovers
 
 
 Diffs
 -
 
   
 core/src/main/java/org/apache/accumulo/core/client/impl/ReplicationOperationsImpl.java
  6a5c74a 
   core/src/main/java/org/apache/accumulo/core/conf/Property.java 2403915 
   core/src/main/java/org/apache/accumulo/core/metadata/RootTable.java 24148b1 
   
 core/src/main/java/org/apache/accumulo/core/metadata/schema/MetadataSchema.java
  534dd7f 
   core/src/main/java/org/apache/accumulo/core/tabletserver/log/LogEntry.java 
 25d0f32 
   
 core/src/test/java/org/apache/accumulo/core/metadata/MetadataTableSchemaTest.java
  PRE-CREATION 
   server/base/src/main/java/org/apache/accumulo/server/fs/VolumeUtil.java 
 fc26ca4 
   server/base/src/main/java/org/apache/accumulo/server/init/Initialize.java 
 e5bdded 
   
 server/base/src/main/java/org/apache/accumulo/server/master/state/MetaDataStateStore.java
  7ee6f0c 
   
 server/base/src/main/java/org/apache/accumulo/server/master/state/MetaDataTableScanner.java
  9be5a67 
   
 server/base/src/main/java/org/apache/accumulo/server/master/state/TabletLocationState.java
  b24b562 
   
 server/base/src/main/java/org/apache/accumulo/server/master/state/TabletStateStore.java
  5413e31 
   
 server/base/src/main/java/org/apache/accumulo/server/master/state/ZooTabletStateStore.java
  ab99396 
   
 server/base/src/main/java/org/apache/accumulo/server/replication/StatusUtil.java
  898e3d4 
   
 server/base/src/main/java/org/apache/accumulo/server/util/ListVolumesUsed.java
  e90d1dd 
   
 server/base/src/main/java/org/apache/accumulo/server/util/MasterMetadataUtil.java
  4ca2d64 
   
 server/base/src/main/java/org/apache/accumulo/server/util/MetadataTableUtil.java
  10cd749 
   
 server/base/src/main/java/org/apache/accumulo/server/util/ReplicationTableUtil.java
  af02a8d 
   
 server/base/src/test/java/org/apache/accumulo/server/util/ReplicationTableUtilTest.java
  b1010c2 
   
 server/gc/src/main/java/org/apache/accumulo/gc/GarbageCollectWriteAheadLogs.java
  1735c0d 
   server/gc/src/main/java/org/apache/accumulo/gc/SimpleGarbageCollector.java 
 c8d5cd6 
   
 server/gc/src/main/java/org/apache/accumulo/gc/replication/CloseWriteAheadLogReferences.java
  3a32727 
   
 server/gc/src/test/java/org/apache/accumulo/gc/GarbageCollectWriteAheadLogsTest.java
  5801faa 
   
 server/gc/src/test/java/org/apache/accumulo/gc/replication/CloseWriteAheadLogReferencesTest.java
  23db83a 
   server/master/src/main/java/org/apache/accumulo/master

Re: Review Request 32224: ACCUMULO-3423

[Eric Newton]

/accumulo/test/MissingWalHeaderCompletesRecoveryIT.java
 b78a311 
  test/src/test/java/org/apache/accumulo/test/NoMutationRecoveryIT.java 6a9975c 
  test/src/test/java/org/apache/accumulo/test/ShellServerIT.java 56a6a70 
  test/src/test/java/org/apache/accumulo/test/functional/WALSunnyDayIT.java 
PRE-CREATION 
  
test/src/test/java/org/apache/accumulo/test/functional/WatchTheWatchCountIT.java
 bd0555b 
  
test/src/test/java/org/apache/accumulo/test/performance/RollWALPerformanceIT.java
 PRE-CREATION 
  
test/src/test/java/org/apache/accumulo/test/replication/GarbageCollectorCommunicatesWithTServersIT.java
 5b89d9c 
  
test/src/test/java/org/apache/accumulo/test/replication/MultiInstanceReplicationIT.java
 9dec16e 
  test/src/test/java/org/apache/accumulo/test/replication/ReplicationIT.java 
54348db 

Diff: https://reviews.apache.org/r/32224/diff/


Testing
---

Ran all tests, except RandomWalk.


Thanks,

Eric Newton

Re: Review Request 32224: ACCUMULO-3423

[Eric Newton]

I know this thing is huge.  Josh has already been over it a couple of times.

I plan to push it to master very soon.  If you would like to hold off for a
more intensive review, please let me know.

At this point, I've run the tests so many times, the ITs are more stable in
my branch than in master.

You can also find this at github:

https://github.com/ericnewton/accumulo-3423

-Eric


On Wed, Mar 18, 2015 at 7:58 PM, Eric Newton eric.new...@gmail.com wrote:


 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/32224/
 ---

 Review request for accumulo.


 Repository: accumulo


 Description
 ---

 Faster WAL rollovers


 Diffs
 -


 core/src/main/java/org/apache/accumulo/core/client/impl/ReplicationOperationsImpl.java
 6a5c74a
   core/src/main/java/org/apache/accumulo/core/conf/Property.java 2403915
   core/src/main/java/org/apache/accumulo/core/metadata/RootTable.java
 24148b1

 core/src/main/java/org/apache/accumulo/core/metadata/schema/MetadataSchema.java
 534dd7f

 core/src/main/java/org/apache/accumulo/core/tabletserver/log/LogEntry.java
 25d0f32

 core/src/test/java/org/apache/accumulo/core/metadata/MetadataTableSchemaTest.java
 PRE-CREATION
   server/base/src/main/java/org/apache/accumulo/server/fs/VolumeUtil.java
 fc26ca4

 server/base/src/main/java/org/apache/accumulo/server/init/Initialize.java
 e5bdded

 server/base/src/main/java/org/apache/accumulo/server/master/state/MetaDataStateStore.java
 7ee6f0c

 server/base/src/main/java/org/apache/accumulo/server/master/state/MetaDataTableScanner.java
 9be5a67

 server/base/src/main/java/org/apache/accumulo/server/master/state/TabletLocationState.java
 b24b562

 server/base/src/main/java/org/apache/accumulo/server/master/state/TabletStateStore.java
 5413e31

 server/base/src/main/java/org/apache/accumulo/server/master/state/ZooTabletStateStore.java
 ab99396

 server/base/src/main/java/org/apache/accumulo/server/replication/StatusUtil.java
 898e3d4

 server/base/src/main/java/org/apache/accumulo/server/util/ListVolumesUsed.java
 e90d1dd

 server/base/src/main/java/org/apache/accumulo/server/util/MasterMetadataUtil.java
 4ca2d64

 server/base/src/main/java/org/apache/accumulo/server/util/MetadataTableUtil.java
 10cd749

 server/base/src/main/java/org/apache/accumulo/server/util/ReplicationTableUtil.java
 af02a8d

 server/base/src/test/java/org/apache/accumulo/server/util/ReplicationTableUtilTest.java
 b1010c2

 server/gc/src/main/java/org/apache/accumulo/gc/GarbageCollectWriteAheadLogs.java
 1735c0d

 server/gc/src/main/java/org/apache/accumulo/gc/SimpleGarbageCollector.java
 c8d5cd6

 server/gc/src/main/java/org/apache/accumulo/gc/replication/CloseWriteAheadLogReferences.java
 3a32727

 server/gc/src/test/java/org/apache/accumulo/gc/GarbageCollectWriteAheadLogsTest.java
 5801faa

 server/gc/src/test/java/org/apache/accumulo/gc/replication/CloseWriteAheadLogReferencesTest.java
 23db83a
   server/master/src/main/java/org/apache/accumulo/master/Master.java
 3762f32

 server/master/src/main/java/org/apache/accumulo/master/MasterClientServiceHandler.java
 f73c236

 server/master/src/main/java/org/apache/accumulo/master/TabletGroupWatcher.java
 d097d75

 server/master/src/main/java/org/apache/accumulo/master/replication/WorkMaker.java
 8532e1b

 server/master/src/main/java/org/apache/accumulo/master/state/MergeStats.java
 44f229e

 server/master/src/test/java/org/apache/accumulo/master/ReplicationOperationsImplTest.java
 a127dcd

 server/master/src/test/java/org/apache/accumulo/master/TestMergeState.java
 b0240f1

 server/master/src/test/java/org/apache/accumulo/master/state/RootTabletStateStoreTest.java
 abceae4
   server/tserver/src/main/findbugs/exclude-filter.xml 47dd1f5

 server/tserver/src/main/java/org/apache/accumulo/server/GarbageCollectionLogger.java
 8f3785e

 server/tserver/src/main/java/org/apache/accumulo/tserver/TabletLevel.java
 PRE-CREATION

 server/tserver/src/main/java/org/apache/accumulo/tserver/TabletServer.java
 662ee31

 server/tserver/src/main/java/org/apache/accumulo/tserver/log/DfsLogger.java
 5acf5eb

 server/tserver/src/main/java/org/apache/accumulo/tserver/log/SortedLogRecovery.java
 c4d9fab

 server/tserver/src/main/java/org/apache/accumulo/tserver/log/TabletServerLogger.java
 711c497

 server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/CommitSession.java
 b4814e4

 server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/DatafileManager.java
 594d9c5

 server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/Tablet.java
 6152500

 server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/TabletCommitter.java
 4bc05a6

 test/src/main/java/org/apache/accumulo/test/performance/thrift/NullTserver.java
 b429607
   test/src/test/java/org/apache/accumulo/proxy/ProxyDurabilityIT.java
 404a8fd

 test/src/test/java/org/apache/accumulo/test

Re: Review Request 30470: Drop the Locator cache every hour

[Eric Newton]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30470/
---

(Updated Feb. 4, 2015, 3:16 p.m.)


Review request for accumulo.


Bugs: ACCUMULO-3549
https://issues.apache.org/jira/browse/ACCUMULO-3549


Repository: accumulo


Description
---

ACCUMULO-3549


Diffs
-

  server/tserver/src/main/java/org/apache/accumulo/tserver/TabletServer.java 
dfec999 

Diff: https://reviews.apache.org/r/30470/diff/


Testing
---


Thanks,

Eric Newton

Re: Concern regarding cache (ACCUMULO-3549 and ACCUMULO-3547) in Apache Accumulo 1.6.2 RC3

[Eric Newton]

We're working ACCUMULO-3549, and a pretty conservative fix will be
committed Monday.


On Sat, Jan 31, 2015 at 12:48 PM, Josh Elser josh.el...@gmail.com wrote:

 That's a good point, Ed, and there hasn't been any other discussion (on
 the mailing lists) so you did the right thing bringing this up here.

 There is no user administration or monitoring support that would allow
 user intervention (aside from restarting a tserver which is a no-go). If
 we're going to include it, like it appears so, we need to both make sure
 that the cache is bounded in size and we have as many people as possible
 look at it (since it's such a late addition to the release -- it's common
 for us to only notice subtleties weeks to months after a change is made
 during normal development cycles).


 Ed Coleman wrote:

 Eric commented on the vote for RC3:

 - - - -
 It would be nice to have ACCUMULO-3547https://issues.
 apache.org/jira/browse/ACCUMULO-3549  in 1.6.2.

 We are running at scale with it at the moment, and it has made a huge
 improvement.  I hate to hold up 1.6.2, though.  If it doesn't make it,
 please update the ticket to point to 1.6.3.
 - - - -

 I generally agree with this and it seems that ACCUMULO-3547 will make it
 into 1.6.2 - which I think is the preferable option. My concerns deal with
 not having ACCUMULO-3549 included in 1.6.2 too.

 In ACCUMULO-3549 Keith made the assumption that end rows are 10 bytes -
 I'm not sure this is a good assumption. If end rows are larger than 10
 bytes, then how much more memory will be required over time? How much
 faster will it grow?

 Without ACCUMULO-3549, what are my options for monitoring / correcting
 the situation if the cache grows too large? Will tablet server performance
 slowly degrade over time because the cache keeps growing?  What will users
 need to do to monitor and then correct this? Will we be in a situation
 where tserevrs will start to run out of memory, we will increase the memory
 allocation if we can, and just kick the can down the road a little further
 and performance will just keep degrading?

 Is there a way to trigger the cache to clear short of restarting a
 tserver? While not optimal, having a utility / script that slowly walks
 across the tservers and clears the cache so that each tserver cache is
 cleared every 12, 24, 48,... hours may be a bridge until ACCUMULO-3549 is
 resolved. If this is the case, it would seem that having the fix in 1.6.3
 would also be a priority.

 Maybe this has been discussed and resolved, but I want to bring this up
 to ensure that the ramifications have been considered and that there is a
 viable mitigation strategy that is communicated to the users. Sorry for the
 doom - end of the world tone I was just trying to emphasis the worst case
 scenarios that I could envision. I think ACCUMULO-3547 is an important
 (even necessary improvement) and I'm not suggesting that it be removed - I
 just want to make sure that I understand the other side effects and know
 our options.

 Ed Coleman

Re: Review Request 30236: Reorganize README

[Eric Newton]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30236/#review70473
---



INSTALL.md
https://reviews.apache.org/r/30236/#comment115689

I like Unpack over Untar



INSTALL.md
https://reviews.apache.org/r/30236/#comment115691

tracer? The default file has localhost which has been a source of 
confusion on a newbie cluster.


- Eric Newton


On Jan. 31, 2015, 12:55 a.m., kturner wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/30236/
 ---
 
 (Updated Jan. 31, 2015, 12:55 a.m.)
 
 
 Review request for accumulo.
 
 
 Bugs: ACCUMULO-1515
 https://issues.apache.org/jira/browse/ACCUMULO-1515
 
 
 Repository: accumulo
 
 
 Description
 ---
 
 Reorganized information in README and converted to markdown.  
 
 At this point I like the INSTALL.md document, but do not really like the 
 content of the README.md ATM.  Putting this up for review to get suggestions.
 
 See how the markdown looks on GH : 
 https://github.com/keith-turner/accumulo/tree/ACCUMULO-1515
 
 
 Diffs
 -
 
   INSTALL.md PRE-CREATION 
   NOTICE af212c2 
   README 4ebb078 
   README.md PRE-CREATION 
   TESTING cf2afba 
   TESTING.md PRE-CREATION 
   assemble/src/main/assemblies/component.xml 3f18da3 
 
 Diff: https://reviews.apache.org/r/30236/diff/
 
 
 Testing
 ---
 
 
 Thanks,
 
 kturner
 

Review Request 30470: Drop the Locator cache every hour

[Eric Newton]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30470/
---

Review request for accumulo.


Repository: accumulo


Description
---

ACCUMULO-3549


Diffs
-

  server/tserver/src/main/java/org/apache/accumulo/tserver/TabletServer.java 
dfec999 

Diff: https://reviews.apache.org/r/30470/diff/


Testing
---


Thanks,

Eric Newton

Re: [VOTE] Apache Accumulo 1.6.2 RC3

[Eric Newton]

-0

It would be nice to have ACCUMULO-3547
https://issues.apache.org/jira/browse/ACCUMULO-3549 in 1.6.2.

We are running at scale with it at the moment, and it has made a huge
improvement.  I hate to hold up 1.6.2, though.  If it doesn't make it,
please update the ticket to point to 1.6.3.

Corey, thanks for all your effort.

-Eric

On Fri, Jan 30, 2015 at 10:36 AM, Keith Turner ke...@deenlo.com wrote:

 On Thu, Jan 29, 2015 at 7:27 PM, Corey Nolet cjno...@gmail.com wrote:

   However I am seeing ACCUMULO-3545[1] that
  I need to investigate.
 
  Ok. I'll cut another RC as soon as that's complete.
 

 Verification completed.   Successfully wrote and verified 31B entries on a
 20 nodes EC2 cluster.   Used Hadoop 2.6.0, ZK 3.4.5, Centos 6, and openjdk
 7.


 
  On Thu, Jan 29, 2015 at 6:03 PM, Josh Elser josh.el...@gmail.com
 wrote:
 
   Given the stuff Keith found already, I'm -1, but I did take some time
  this
   RC to rerun some tests. I had one IT that failed on me from the source
   build which we can fix later -- things are looking good otherwise from
 my
   testing.
  
   Thanks for working through this Corey, and Keith for finding bugs :)
  
  
   Corey Nolet wrote:
  
  Devs,
  
Please consider the following candidate for Apache Accumulo 1.6.2
  
Branch: 1.6.2-rc3
SHA1: 3a6987470c1e5090a2ca159614a80f0fa50393bf
Staging Repository:
   https://repository.apache.org/content/repositories/
   orgapacheaccumulo-1021/
  
Source tarball:
   https://repository.apache.org/content/repositories/
   orgapacheaccumulo-1021/org/apache/accumulo/accumulo/1.6.
   2/accumulo-1.6.2-src.tar.gz
Binary tarball:
   https://repository.apache.org/content/repositories/
   orgapacheaccumulo-1021/org/apache/accumulo/accumulo/1.6.
   2/accumulo-1.6.2-bin.tar.gz
(Append .sha1, .md5 or .asc to download the signature/hash
  for
   a
   given artifact.)
  
Signing keys available at:
  https://www.apache.org/dist/accumulo/KEYS
  
Over 1.6.1, we have 148 issues resolved:
   https://git-wip-us.apache.org/repos/asf?p=accumulo.git;a=
   blob_plain;f=CHANGES;hb=1.6.2-rc3
  
Testing: All unit, integration and functional tests are passing.
  
API compatibility report for 1.6.1 to 1.6.2:
   http://people.apache.org/~cjnolet/accumulo-1.6.2-rc3/
   compat_reports/accumulo/1.6.1_to_1.6.2/compat_report.html
  
API backwards compatibility report for 1.6.2 to 1.6.1:
   http://people.apache.org/~cjnolet/accumulo-1.6.2-rc3/
   compat_reports/accumulo/1.6.2_to_1.6.1/compat_report.html
  
The vote will be open until Saturday, January 31st 12:00AM UTC
  (1/30
   8:00PM ET, 1/30 5:00PM PT)
  
  
 

Re: Review Request 30470: Drop the Locator cache every hour

[Eric Newton]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30470/
---

(Updated Jan. 31, 2015, 12:33 a.m.)


Review request for accumulo.


Repository: accumulo


Description
---

ACCUMULO-3549


Diffs (updated)
-

  server/tserver/src/main/java/org/apache/accumulo/tserver/TabletServer.java 
dfec999 

Diff: https://reviews.apache.org/r/30470/diff/


Testing
---


Thanks,

Eric Newton

Re: Review Request 30417: ACCUMULO-3462 ACCUMULO-3541 fixed compaction logging and state bug

[Eric Newton]

 On Jan. 29, 2015, 5:54 p.m., kturner wrote:
  server/tserver/src/main/java/org/apache/accumulo/tserver/Tablet.java, line 
  3347
  https://reviews.apache.org/r/30417/diff/1/?file=840136#file840136line3347
 
  While trying apply this patch to 1.5 I noticed that 1.5 is different.  
  1.5 does not keep a reference to what `Trace.on()` returns.  
  
  If this reference does not need to be kept, then the fix may be 
  simpler.  Eric do you know if we need to keep the reference, or can we just 
  call `curr.stop()` later insread of `span.stop()`.

They should be equivalent.


- Eric


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30417/#review70234
---


On Jan. 29, 2015, 3:59 p.m., kturner wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/30417/
 ---
 
 (Updated Jan. 29, 2015, 3:59 p.m.)
 
 
 Review request for accumulo and Eric Newton.
 
 
 Bugs: ACCUMULO-3462 and ACCUMULO-3541
 https://issues.apache.org/jira/browse/ACCUMULO-3462
 https://issues.apache.org/jira/browse/ACCUMULO-3541
 
 
 Repository: accumulo
 
 
 Description
 ---
 
 Fixes compaction logging bug and bug with compaction state not being reset.  
 Patch builds on work already committed for ACCUMULO-3462.
 
 I am slightly nervous about the changes w/ control flow, but I feel this is 
 the best solution to avoid duplicate logging.
 
 
 Diffs
 -
 
   server/tserver/src/main/java/org/apache/accumulo/tserver/Tablet.java 
 7420ec4 
 
 Diff: https://reviews.apache.org/r/30417/diff/
 
 
 Testing
 ---
 
 Still running ITs.  Getting this patch out for review so we can move forward 
 for 1.6.2.
 
 
 Thanks,
 
 kturner
 

Re: Review Request 30417: ACCUMULO-3462 ACCUMULO-3541 fixed compaction logging and state bug

[Eric Newton]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30417/#review70211
---

Ship it!


Ship It!

- Eric Newton


On Jan. 29, 2015, 3:59 p.m., kturner wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/30417/
 ---
 
 (Updated Jan. 29, 2015, 3:59 p.m.)
 
 
 Review request for accumulo and Eric Newton.
 
 
 Bugs: ACCUMULO-3462 and ACCUMULO-3541
 https://issues.apache.org/jira/browse/ACCUMULO-3462
 https://issues.apache.org/jira/browse/ACCUMULO-3541
 
 
 Repository: accumulo
 
 
 Description
 ---
 
 Fixes compaction logging bug and bug with compaction state not being reset.  
 Patch builds on work already committed for ACCUMULO-3462.
 
 I am slightly nervous about the changes w/ control flow, but I feel this is 
 the best solution to avoid duplicate logging.
 
 
 Diffs
 -
 
   server/tserver/src/main/java/org/apache/accumulo/tserver/Tablet.java 
 7420ec4 
 
 Diff: https://reviews.apache.org/r/30417/diff/
 
 
 Testing
 ---
 
 Still running ITs.  Getting this patch out for review so we can move forward 
 for 1.6.2.
 
 
 Thanks,
 
 kturner
 

Re: Review Request 30236: Reorganize README

[Eric Newton]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30236/#review69816
---



README.md
https://reviews.apache.org/r/30236/#comment114622

in-depth



README.md
https://reviews.apache.org/r/30236/#comment114623

delete Composed of example 



TESTING.md
https://reviews.apache.org/r/30236/#comment114627

awkward phrasing very encompassing of checking for; recommend check for


- Eric Newton


On Jan. 27, 2015, 4:13 p.m., kturner wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/30236/
 ---
 
 (Updated Jan. 27, 2015, 4:13 p.m.)
 
 
 Review request for accumulo.
 
 
 Bugs: ACCUMULO-1515
 https://issues.apache.org/jira/browse/ACCUMULO-1515
 
 
 Repository: accumulo
 
 
 Description
 ---
 
 Reorganized information in README and converted to markdown.  
 
 At this point I like the INSTALL.md document, but do not really like the 
 content of the README.md ATM.  Putting this up for review to get suggestions.
 
 See how the markdown looks on GH : 
 https://github.com/keith-turner/accumulo/tree/ACCUMULO-1515
 
 
 Diffs
 -
 
   BUILD.md PRE-CREATION 
   INSTALL.md PRE-CREATION 
   NOTICE af212c2 
   README 4ebb078 
   README.md PRE-CREATION 
   TESTING cf2afba 
   TESTING.md PRE-CREATION 
   assemble/src/main/assemblies/component.xml 3f18da3 
 
 Diff: https://reviews.apache.org/r/30236/diff/
 
 
 Testing
 ---
 
 
 Thanks,
 
 kturner
 

Re: Re: accumulo query delete

[Eric Newton]

Yes, it will most likely cause confusion, even if it is consistent.  Just
like creating and deleting a table from different threads would cause a
confusing situation, even if it works correctly.

Example:
Thread A adds write permission for User1 to table X
Thread B removes write permission for User1 to table X

Now, what should the write permission be for User1 on table X?  It cannot
be determined, because the order the permissions were run cannot be known.
However, it should be safe, in that they are executed atomically, and the
updates are sent in the same order to all the servers, resulting in
eventual agreement (in a few seconds).

-Eric


On Fri, Jan 23, 2015 at 1:49 AM, panqing...@163.com panqing...@163.com
wrote:

   HI ,thank you for this eamil,  My question is if multiple threads
 modifying user permissions at the same time, will lead to a problem? For
 example, a users will need to admin, B users will need to root, assuming
 the two at the same time, will lead to a problem.



 panqing...@163.com

 From: Eric Newton [via Apache Accumulo]
 Date: 2015-01-22 00:32
 To: panqing...@163.com
 Subject: Re: accumulo query delete
 Ah. OK.

 All users' permissions and Authorizations are stored in zookeeper and
 update servers asynchronously. The assumption is that these operations are
 not needed to be fast and atomic: they are performed infrequently. That is,
 a few seconds update delay is acceptable, caching the data is good.

 If you need row-level atomic updates, please look into ConditionalWriter.
 In some cases, an IsolatedScanner may work better than the
 WholeRowIterator, though for small records, the WholeRowIterator works
 fine.

 You probably want the priority of the WholeRowIterator to be higher than
 the VersioningIterator, so that you do not get multiple versions for a
 particular cell of the table.

 -Eric

 On Wed, Jan 21, 2015 at 3:12 AM, [hidden email] [hidden email]
 wrote:

  Thank your  reply this question.
 1、  public void deleteUser(String licenseKey, String userId) throws
  AccumuloException {
  Connector conn = AccumuloClientSingleton.INSTANCE.getConnector();
  BatchDeleter bd = null;
  try {
 
 
 
 conn.securityOperations().changeUserAuthorizations(AccumuloConstants.USER_NAME,
  new Authorizations(licenseKey));
  bd = conn.createBatchDeleter(AccumuloConstants.XT_USER, new
  Authorizations(licenseKey), 10,
  new BatchWriterConfig());
  SetRange ranges = new HashSetRange();
  ranges.add(new Range(userId));
  bd.setRanges(ranges);
  bd.delete();
  } catch (Exception e) {
  throw new AccumuloException(删除用户信息时异常, e);
  } finally {
  if (bd != null)
  bd.close();
  }
  }
  2、  public XtUser getUser(String licenseKey, String userAccount) throws
  AccumuloException {
  Connector conn = AccumuloClientSingleton.INSTANCE.getConnector();
  MapString, XtUser map = new HashMapString, XtUser();
  try {
 
 
 
 conn.securityOperations().changeUserAuthorizations(AccumuloConstants.USER_NAME,
  new Authorizations(licenseKey));
  Scanner scanner =
  conn.createScanner(AccumuloConstants.USER_NAME, new
  Authorizations(licenseKey));
  // 行迭代器
  IteratorSetting it = new IteratorSetting(1,
 WholeRowIterator,
  WholeRowIterator.class);
  scanner.addScanIterator(it);
  for (EntryKey, Value entry : scanner) {
  XtUser xtUser = new XtUser();
  for (EntryKey, Value actualEntry :
  WholeRowIterator.decodeRow(entry.getKey(), entry.getValue())
  .entrySet()) {
  String qualifier =
  actualEntry.getKey().getColumnFamily().toString();
  String value = actualEntry.getValue().toString();
  if (qualifier.equals(role_id)) {
  xtUser.setRoleId(value);
  } else if (qualifier.equals(role_name)) {
  xtUser.setRoleName(role_name);
  } else if (qualifier.equals(user_account)) {
  xtUser.setUserAccount(value);
  }
  map.put(xtUser.getUserAccount(), xtUser);
  }
  }
  } catch (Exception e) {
  throw new AccumuloException(获取用户消息异常, e);
  }
  return map.get(userAccount);
  }
 
  Method 1, method 2 will modify the user's permission, if concurrent case,
  is
  also the method 1, method 2 and is called, should be how to deal with?
 
 
 
  --
  View this message in context:
 
 http://apache-accumulo.1065345.n5.nabble.com/accumulo-query-delete-tp12965p12969.html
  Sent from the Developers mailing list archive at Nabble.com.
 




 If you reply to this email, your message will be added to the discussion
 below

Re: accumulo query delete

[Eric Newton]

I'm having some difficulty understanding the question.

Can you provide an example using java or just the accumulo shell to
demonstrate the question?

-Eric

On Tue, Jan 20, 2015 at 9:58 PM, panqing...@163.com panqing...@163.com
wrote:

 HI
   Recently the use of accumulo to do the user information storage, a
 tenant contains a plurality of user, the user can only query the current
 tenants of the data, set the Visbility for the tenants, then the query ID,
 according to each tenant ID query user information, but for every need to
 query modify permissions (setauths), when the tenants under different user,
 query, delete all need to modify permissions, how to ensure the
 synchronous? For example,:a tenant Join， b tenant  Jack , and jack
 operation at the same time, the join Jack query to delete user information,
 user information, delete the former setauths = a ,query before setauts=b
 when the concurrent situation should be how to deal with?



 panqing...@163.com

Re: [VOTE][LAZY] Format all supported branches

[Eric Newton]

+0

I don't think it's worth the disruption, but I don't mind if you're going
to do all the work.


On Wed, Jan 7, 2015 at 3:47 PM, Mike Drob mad...@cloudera.com wrote:

 Also, if you're using Eclipse to do the auto-format, please check for
 trailing white-space on otherwise empty javadoc lines.

 If you automate this in some fashion outside of Eclipse (because other
 people may prefer other editors), this would be a useful script to add to a
 top-level dev-tools folder.

 On Wed, Jan 7, 2015 at 2:36 PM, David Medinets david.medin...@gmail.com
 wrote:

  +1
 
  Are you automating the process so that you can re-apply the same steps
  in one year?
 
  On Wed, Jan 7, 2015 at 3:31 PM, Christopher ctubb...@apache.org wrote:
   Can do. It's a bit more work for me, because I have to repeat the same
   actions over and over again, but if it helps history look a little
  cleaner,
   i can do it, and just stick to -sours and repeat for the next branch..
  
  
   --
   Christopher L Tubbs II
   http://gravatar.com/ctubbsii
  
   On Wed, Jan 7, 2015 at 3:25 PM, Mike Drob mad...@cloudera.com wrote:
  
   Please do not do formatting during merge conflict resolution, and make
   those be separate commits.
  
   On Wed, Jan 7, 2015 at 2:23 PM, Josh Elser josh.el...@gmail.com
  wrote:
  
ack'ed
   
   
John Vines wrote:
   
+1
   
On Wed, Jan 7, 2015 at 3:12 PM, Christopherctubb...@apache.org
   wrote:
   
 To make it easier to apply some minimal checkstyle rules for
ACCUMULO-3451,
I'm announcing my intentions to do a full, one-time, auto-format
 and
organize imports on all our supported branches (1.5, 1.6, and
  master)
   to
bring us up to some degree of compliance with our agreed-upon
   formatting
standards.
   
Benefits:
To have additional checks, in particular against javadoc problems
  and
other
common trivial warnings in the build.
To ensure less divergence from our agreed-upon formatting
 standards.
Formatting first makes it much less tedious and easier on me to
 add
   these
checks to the build.
   
Issues I've considered:
I will deal with all the merge conflicts.
I will ignore generated thrift code.
Conflicts with new code in people's branches should be minimal
 (and
easily
resolved by formatting according to our standards).
Regarding concerns about history tracking, in general, each format
   change
is small, but they are numerous. So, the impact on tracking
 history
should
be very minimal (you'll see things like a brace moved to the same
  line
   as
the else statement it is associated with... stuff that won't
  generally
affect your ability to debug).
I'll also do a format only commit, separately from any
 substantive
changes regarding the rule changes, so the mass formatting change
  will
happen in one place, and it will also be easy to revert, if
  absolutely
necessary.
   
I'll give this 24 hours (it can be reverted if somebody objects
  after
that).
   
--
Christopher L Tubbs II
http://gravatar.com/ctubbsii
   
   
   
  
 

Re: Review Request 29230: ACCUMULO-3439 Add RegexGroupBalancer

[Eric Newton]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29230/#review66915
---


Nit: whitespace
Nit: single-line statements w/out braces


server/base/src/main/java/org/apache/accumulo/server/master/balancer/GroupBalancer.java
https://reviews.apache.org/r/29230/#comment110619

final



server/base/src/main/java/org/apache/accumulo/server/master/balancer/GroupBalancer.java
https://reviews.apache.org/r/29230/#comment110620

this(null);



server/base/src/main/java/org/apache/accumulo/server/master/balancer/GroupBalancer.java
https://reviews.apache.org/r/29230/#comment110621

final



server/base/src/main/java/org/apache/accumulo/server/master/balancer/GroupBalancer.java
https://reviews.apache.org/r/29230/#comment110622

final



server/base/src/main/java/org/apache/accumulo/server/master/balancer/GroupBalancer.java
https://reviews.apache.org/r/29230/#comment110623

final



server/base/src/main/java/org/apache/accumulo/server/master/balancer/GroupBalancer.java
https://reviews.apache.org/r/29230/#comment110624

final



server/base/src/main/java/org/apache/accumulo/server/master/balancer/GroupBalancer.java
https://reviews.apache.org/r/29230/#comment110625

@Override



server/base/src/main/java/org/apache/accumulo/server/master/balancer/GroupBalancer.java
https://reviews.apache.org/r/29230/#comment110626

@Override



server/base/src/main/java/org/apache/accumulo/server/master/balancer/GroupBalancer.java
https://reviews.apache.org/r/29230/#comment110627

@Override



server/base/src/main/java/org/apache/accumulo/server/master/balancer/GroupBalancer.java
https://reviews.apache.org/r/29230/#comment110628

Could this class be made static by passing in the table ID?


- Eric Newton


On Dec. 19, 2014, 1:52 a.m., kturner wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/29230/
 ---
 
 (Updated Dec. 19, 2014, 1:52 a.m.)
 
 
 Review request for accumulo.
 
 
 Bugs: ACCUMULO-3439
 https://issues.apache.org/jira/browse/ACCUMULO-3439
 
 
 Repository: accumulo
 
 
 Description
 ---
 
 The patch contains a new balancer, test, and documentation.
 
 
 Diffs
 -
 
   docs/src/main/resources/examples/README.rgbalancer PRE-CREATION 
   
 server/base/src/main/java/org/apache/accumulo/server/master/balancer/GroupBalancer.java
  PRE-CREATION 
   
 server/base/src/main/java/org/apache/accumulo/server/master/balancer/RegexGroupBalancer.java
  PRE-CREATION 
   
 server/base/src/main/java/org/apache/accumulo/server/master/balancer/TableLoadBalancer.java
  5eae890 
   
 server/base/src/test/java/org/apache/accumulo/server/master/balancer/GroupBalancerTest.java
  PRE-CREATION 
   
 test/src/test/java/org/apache/accumulo/test/functional/RegexGroupBalanceIT.java
  PRE-CREATION 
 
 Diff: https://reviews.apache.org/r/29230/diff/
 
 
 Testing
 ---
 
 The new unit test and intergeration test added in the patch run.  
 
 
 Thanks,
 
 kturner
 

Re: 1.6.2 candidates

[Eric Newton]

We are running 1.6.1 w/patches in production already.  I would much rather
have a 1.6.2 official release.

I may have temporary access to a small cluster (3-ish racks) to run some of
the long running tests on bare metal.

Testing sooner, rather than later is preferable.




On Tue, Dec 16, 2014 at 7:18 PM, Corey Nolet cjno...@gmail.com wrote:

 I have cycles to spin the RCs- I wouldn't mind finishing the updates (per
 my notes) of the release documentation as well.

 On Tue, Dec 16, 2014 at 7:11 PM, Christopher ctubb...@apache.org wrote:
 
  I think it'd be good to let somebody else exercise the process a bit,
 but I
  can make the RCs if nobody else volunteers. My primary concern is that
  people will have time to test.
 
 
  --
  Christopher L Tubbs II
  http://gravatar.com/ctubbsii
 
  On Tue, Dec 16, 2014 at 6:37 PM, Josh Elser josh.el...@gmail.com
 wrote:
  
   +1 There are lots of good bug fixes in 1.6.2 already.
  
   I can make some time to test, document, etc. Are you volunteering to
 spin
   the RCs as well?
  
  
   Christopher wrote:
  
   I'm thinking we should look at releasing 1.6.2 in January. I'd say
  sooner,
   but I don't know if people will have time to test if we start putting
   together RCs this week or next.
  
   --
   Christopher L Tubbs II
   http://gravatar.com/ctubbsii
  
  
 

Re: [VOTE] adoption of semver

[Eric Newton]

+1

On Tue, Dec 9, 2014 at 1:23 PM, Billie Rinaldi bil...@apache.org wrote:

 I would like to call a vote on adopting semantic versioning (
 http://semver.org/) for future releases.

 This vote is subject to majority approval and will remain open for 72
 hours.

 +1: Adopt semantic versioning for all future releases
 +0: Don't care
 -1: Do not adopt semantic versioning because ...

 Here is my +1.

 Billie

Re: Review Request 26572: ACCUMULO-898

[Eric Newton]

,

Eric Newton

Re: Review Request 27829: ACCUMULO-1085

[Eric Newton]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27829/
---

(Updated Nov. 24, 2014, 1:53 p.m.)


Review request for accumulo.


Changes
---

Add link to JIRA issue.


Bugs: ACCUMULO-1085
https://issues.apache.org/jira/browse/ACCUMULO-1085


Repository: accumulo


Description
---

Allow for multiple threads to perform assignment, but use only a single thread 
for recovery playback


Diffs
-

  core/src/main/java/org/apache/accumulo/core/conf/Property.java f59b654 
  server/tserver/src/main/java/org/apache/accumulo/tserver/TabletServer.java 
1e81947 
  
server/tserver/src/main/java/org/apache/accumulo/tserver/TabletServerResourceManager.java
 ba86522 
  server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/Tablet.java 
9490903 
  test/src/test/java/org/apache/accumulo/test/AssignmentThreadsIT.java 
PRE-CREATION 

Diff: https://reviews.apache.org/r/27829/diff/


Testing
---

Added an IT that verifies that tablet loading goes faster with more threads.


Thanks,

Eric Newton

Re: Review Request 27829: ACCUMULO-1085

[Eric Newton]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27829/
---

(Updated Nov. 24, 2014, 8:15 p.m.)


Review request for accumulo.


Bugs: ACCUMULO-1085
https://issues.apache.org/jira/browse/ACCUMULO-1085


Repository: accumulo


Description
---

Allow for multiple threads to perform assignment, but use only a single thread 
for recovery playback


Diffs (updated)
-

  core/src/main/java/org/apache/accumulo/core/conf/Property.java f59b654 
  server/tserver/src/main/java/org/apache/accumulo/tserver/TabletServer.java 
1e81947 
  
server/tserver/src/main/java/org/apache/accumulo/tserver/TabletServerResourceManager.java
 ba86522 
  server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/Tablet.java 
9490903 
  test/src/test/java/org/apache/accumulo/test/AssignmentThreadsIT.java 
PRE-CREATION 

Diff: https://reviews.apache.org/r/27829/diff/


Testing
---

Added an IT that verifies that tablet loading goes faster with more threads.


Thanks,

Eric Newton

Re: Review Request 27829: ACCUMULO-1085

[Eric Newton]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27829/
---

(Updated Nov. 24, 2014, 8:33 p.m.)


Review request for accumulo.


Bugs: ACCUMULO-1085
https://issues.apache.org/jira/browse/ACCUMULO-1085


Repository: accumulo


Description
---

Allow for multiple threads to perform assignment, but use only a single thread 
for recovery playback


Diffs (updated)
-

  core/src/main/java/org/apache/accumulo/core/conf/Property.java 1195668 
  server/tserver/src/main/java/org/apache/accumulo/tserver/TabletServer.java 
93161ee 
  
server/tserver/src/main/java/org/apache/accumulo/tserver/TabletServerResourceManager.java
 ba86522 
  test/src/test/java/org/apache/accumulo/test/AssignmentThreadsIT.java 
PRE-CREATION 
  test/src/test/java/org/apache/accumulo/test/VerifySerialRecoveryIT.java 
PRE-CREATION 
  test/src/test/java/org/apache/accumulo/test/functional/BulkFileIT.java 
475d5cf 
  
test/src/test/java/org/apache/accumulo/test/functional/FunctionalTestUtils.java 
1246efe 

Diff: https://reviews.apache.org/r/27829/diff/


Testing
---

Added an IT that verifies that tablet loading goes faster with more threads.


Thanks,

Eric Newton

Review Request 27829: ACCUMULO-1085

[Eric Newton]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27829/
---

Review request for accumulo.


Repository: accumulo


Description
---

Allow for multiple threads to perform assignment, but use only a single thread 
for recovery playback


Diffs
-

  core/src/main/java/org/apache/accumulo/core/conf/Property.java f59b654 
  server/tserver/src/main/java/org/apache/accumulo/tserver/TabletServer.java 
1e81947 
  
server/tserver/src/main/java/org/apache/accumulo/tserver/TabletServerResourceManager.java
 ba86522 
  server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/Tablet.java 
9490903 
  test/src/test/java/org/apache/accumulo/test/AssignmentThreadsIT.java 
PRE-CREATION 

Diff: https://reviews.apache.org/r/27829/diff/


Testing
---

Added an IT that verifies that tablet loading goes faster with more threads.


Thanks,

Eric Newton

Re: Review Request 27829: ACCUMULO-1085

[Eric Newton]

 On Nov. 10, 2014, 9:38 p.m., Mike Drob wrote:
  test/src/test/java/org/apache/accumulo/test/AssignmentThreadsIT.java, line 
  34
  https://reviews.apache.org/r/27829/diff/1/?file=757270#file757270line34
 
  Plkease add a test that only one recovery will happen at a time.

Any idea how I can do that?


 On Nov. 10, 2014, 9:38 p.m., Mike Drob wrote:
  test/src/test/java/org/apache/accumulo/test/AssignmentThreadsIT.java, line 
  44
  https://reviews.apache.org/r/27829/diff/1/?file=757270#file757270line44
 
  Is this just chars 0-9  a-f?

yes; originally I left the splits as binary, but it was really ugly to read the 
logs.


 On Nov. 10, 2014, 9:38 p.m., Mike Drob wrote:
  test/src/test/java/org/apache/accumulo/test/AssignmentThreadsIT.java, line 
  73
  https://reviews.apache.org/r/27829/diff/1/?file=757270#file757270line73
 
  Why go through arithmetic gymnastics instead of just sleep(11000)?

Taking the tablets offline probably takes most of the 11 seconds.  This just 
extends the wait to make sure that it's at least 11 seconds, but not any more 
than that.


 On Nov. 10, 2014, 9:38 p.m., Mike Drob wrote:
  server/tserver/src/main/java/org/apache/accumulo/tserver/TabletServer.java, 
  line 2904
  https://reviews.apache.org/r/27829/diff/1/?file=757267#file757267line2904
 
  Why are we using a Semaphore over just making the method synchronized? 
  It would make sense if there was some other code section also checking on 
  the semaphore, but I don't see it.

For the fairness property. I'm going to have to make changes in this approach 
because the metadata table has to be able to recover at all times, and the 
minor compaction must be included in the recovery (Thanks to an offline review 
by [~kturner].)


- Eric


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27829/#review60681
---


On Nov. 10, 2014, 9:11 p.m., Eric Newton wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/27829/
 ---
 
 (Updated Nov. 10, 2014, 9:11 p.m.)
 
 
 Review request for accumulo.
 
 
 Repository: accumulo
 
 
 Description
 ---
 
 Allow for multiple threads to perform assignment, but use only a single 
 thread for recovery playback
 
 
 Diffs
 -
 
   core/src/main/java/org/apache/accumulo/core/conf/Property.java f59b654 
   server/tserver/src/main/java/org/apache/accumulo/tserver/TabletServer.java 
 1e81947 
   
 server/tserver/src/main/java/org/apache/accumulo/tserver/TabletServerResourceManager.java
  ba86522 
   server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/Tablet.java 
 9490903 
   test/src/test/java/org/apache/accumulo/test/AssignmentThreadsIT.java 
 PRE-CREATION 
 
 Diff: https://reviews.apache.org/r/27829/diff/
 
 
 Testing
 ---
 
 Added an IT that verifies that tablet loading goes faster with more threads.
 
 
 Thanks,
 
 Eric Newton
 

Re: Review Request 27106: ACCUMULO-898 convert to htrace

[Eric Newton]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27106/#review60318
---

Ship it!


- Eric Newton


On Nov. 6, 2014, 11:20 p.m., Billie Rinaldi wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/27106/
 ---
 
 (Updated Nov. 6, 2014, 11:20 p.m.)
 
 
 Review request for accumulo and Eric Newton.
 
 
 Repository: accumulo
 
 
 Description
 ---
 
 Main diff is ACCUMULO-898-5.patch.  This patch could stand alone.  The file 
 attached does a lot of additional refactoring, removing or deprecating 
 everything in the trace module.  So we can decide whether to apply just the 
 first patch or both.
 
 
 Diffs
 -
 
   assemble/bin/stop-all.sh 4bf06c0 
   assemble/pom.xml 89a3747 
   assemble/src/main/assemblies/component.xml 599d26c 
   core/pom.xml 10e7d71 
   core/src/main/java/org/apache/accumulo/core/client/ClientConfiguration.java 
 39b460d 
   core/src/main/java/org/apache/accumulo/core/conf/Property.java a558760 
   core/src/main/java/org/apache/accumulo/core/conf/PropertyType.java fc20535 
   core/src/main/java/org/apache/accumulo/core/trace/AsyncSpanReceiver.java 
 PRE-CREATION 
   core/src/main/java/org/apache/accumulo/core/trace/DistributedTrace.java 
 83f5c26 
   core/src/main/java/org/apache/accumulo/core/trace/SendSpansViaThrift.java 
 PRE-CREATION 
   core/src/main/java/org/apache/accumulo/core/trace/TraceDump.java b44cc3e 
   core/src/main/java/org/apache/accumulo/core/trace/TraceFormatter.java 
 9d860d9 
   core/src/main/java/org/apache/accumulo/core/trace/ZooTraceClient.java 
 9586eaa 
   core/src/main/java/org/apache/accumulo/core/util/ThriftUtil.java da4e567 
   docs/src/main/asciidoc/chapters/administration.txt d5e73f0 
   docs/src/main/resources/distributedTracing.html 54c9095 
   
 examples/simple/src/main/java/org/apache/accumulo/examples/simple/client/TracingExample.java
  a542263 
   
 minicluster/src/main/java/org/apache/accumulo/minicluster/MiniAccumuloInstance.java
  54897cb 
   pom.xml ebc2f2f 
   server/base/src/main/java/org/apache/accumulo/server/Accumulo.java 5c93a53 
   server/base/src/main/java/org/apache/accumulo/server/init/Initialize.java 
 45e883d 
   
 server/base/src/main/java/org/apache/accumulo/server/trace/TraceFSDataInputStream.java
  5162e01 
   
 server/base/src/main/java/org/apache/accumulo/server/trace/TraceFileSystem.java
  d3fbad7 
   
 server/base/src/main/java/org/apache/accumulo/server/util/AccumuloStatus.java 
 6c7fd47 
   server/base/src/main/java/org/apache/accumulo/server/util/ZooZap.java 
 1f59531 
   server/gc/src/main/java/org/apache/accumulo/gc/SimpleGarbageCollector.java 
 f943ac1 
   server/master/src/main/java/org/apache/accumulo/master/Master.java e3fc69d 
   
 server/master/src/main/java/org/apache/accumulo/master/replication/ReplicationDriver.java
  a52f743 
   server/monitor/src/main/java/org/apache/accumulo/monitor/Monitor.java 
 49bb56d 
   
 server/monitor/src/main/java/org/apache/accumulo/monitor/servlets/trace/ShowTrace.java
  a476201 
   
 server/monitor/src/test/java/org/apache/accumulo/monitor/ShowTraceLinkTypeTest.java
  a630434 
   server/tracer/src/main/java/org/apache/accumulo/tracer/TraceServer.java 
 4858b8a 
   
 server/tserver/src/main/java/org/apache/accumulo/tserver/BulkFailedCopyProcessor.java
  f7bda49 
   server/tserver/src/main/java/org/apache/accumulo/tserver/InMemoryMap.java 
 9a1117d 
   server/tserver/src/main/java/org/apache/accumulo/tserver/TabletServer.java 
 54c75f8 
   
 server/tserver/src/main/java/org/apache/accumulo/tserver/replication/AccumuloReplicaSystem.java
  732907d 
   server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/Tablet.java 
 ef3a0c9 
   shell/src/main/java/org/apache/accumulo/shell/Shell.java a0ff17a 
   shell/src/main/java/org/apache/accumulo/shell/commands/TraceCommand.java 
 7f63570 
   test/src/main/java/org/apache/accumulo/test/TestIngest.java 7f6c514 
   test/src/main/java/org/apache/accumulo/test/VerifyIngest.java 74b03e4 
   test/src/test/java/org/apache/accumulo/test/ConditionalWriterIT.java 
 4d9d479 
   test/src/test/java/org/apache/accumulo/test/functional/BulkFileIT.java 
 80ee990 
   test/src/test/java/org/apache/accumulo/test/functional/ExamplesIT.java 
 210e057 
   test/src/test/java/org/apache/accumulo/test/functional/SimpleMacIT.java 
 03677f4 
   trace/pom.xml aacfb56 
   
 trace/src/main/java/org/apache/accumulo/trace/instrument/CloudtraceSpan.java 
 PRE-CREATION 
   trace/src/main/java/org/apache/accumulo/trace/instrument/CountSampler.java 
 9a5bdbb 
   trace/src/main/java/org/apache/accumulo/trace/instrument/Sampler.java 
 4abb40a 
   trace/src/main/java/org/apache/accumulo/trace/instrument/Span.java 5267174 
   trace/src/main/java/org

Re: Review Request 27654: Add introspection of long running assignments

[Eric Newton]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27654/#review60167
---



core/src/main/java/org/apache/accumulo/core/conf/Property.java
https://reviews.apache.org/r/27654/#comment101511

time a compaction can run ... don't think that's what this does.


- Eric Newton


On Nov. 6, 2014, 12:58 a.m., Josh Elser wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/27654/
 ---
 
 (Updated Nov. 6, 2014, 12:58 a.m.)
 
 
 Review request for accumulo.
 
 
 Bugs: ACCUMULO-3304
 https://issues.apache.org/jira/browse/ACCUMULO-3304
 
 
 Repository: accumulo
 
 
 Description
 ---
 
 Watches assignments and reports when an assignment is running for longer than 
 a configured time.
 
 
 Diffs
 -
 
   core/src/main/java/org/apache/accumulo/core/conf/Property.java 56f3d9c 
   
 server/tserver/src/main/java/org/apache/accumulo/tserver/ActiveAssignmentRunnable.java
  PRE-CREATION 
   
 server/tserver/src/main/java/org/apache/accumulo/tserver/RunnableStartedAt.java
  PRE-CREATION 
   server/tserver/src/main/java/org/apache/accumulo/tserver/TabletServer.java 
 94be0bb 
   
 server/tserver/src/main/java/org/apache/accumulo/tserver/TabletServerResourceManager.java
  935ffeb 
 
 Diff: https://reviews.apache.org/r/27654/diff/
 
 
 Testing
 ---
 
 Very minimal.
 
 
 Thanks,
 
 Josh Elser
 

Re: Review Request 27654: Add introspection of long running assignments

[Eric Newton]

https://issues.apache.org/jira/browse/ACCUMULO-3311



On Thu, Nov 6, 2014 at 7:29 PM, Eric Newton eric.new...@gmail.com wrote:

 It would be nice to model Danger! messages with All Clear! directly.

 I'll make a ticket.

 On Thu, Nov 6, 2014 at 3:47 PM, Josh Elser josh.el...@gmail.com wrote:



  On Nov. 6, 2014, 5:47 p.m., kturner wrote:
  
 server/tserver/src/main/java/org/apache/accumulo/tserver/TabletServerResourceManager.java,
 line 250
   
 https://reviews.apache.org/r/27654/diff/3/?file=751140#file751140line250
  
   The compaction code remembers when it logged an exception and
 does not do it again.   It also logs a message if the compaction becomes
 unstuck.  An advantage I thought of w/ repeatedly logging, is that you
 could see the stack trace changing (or not).
  
  
   The stack trace is  a possible trace.  By the time logging
 happens, the assignment could have completed and the thread could have
 moved on to other things.
 
  Josh Elser wrote:
  Yeah, since these are running fairly regularly (order of seconds) a
 stuck assignment could get really spammy. Like you point out, there could
 be value gained from printing out the stack more than once. Maybe I could
 add some backoff which only warns so often?
 
  bq. By the time logging happens, the assignment could have
 completed and the thread could have moved on to other things.
 
  Do you think the message should be updated to be more clear about
 this? A Maybe you should look into this type message?
 
  kturner wrote:
   a stuck assignment could get really spammy
 
  I think that spam is probably ok as long as the default is high
 enough such that when it does happen, its something to be concerned about.
 Could make the timer check a little less frequently.
 
   Do you think the message should be updated to be more clear about
 this?
 
  I think compaction code just says its a possible stack trace.   I
 suppose a good solution would be to have error codes, then user can look up
 error code and get nitty gritty details.  Can't really put too much info in
 log message.
 
  Josh Elser wrote:
  bq. Could make the timer check a little less frequently.
 
  As long as we have a long threshold for warning about a stuck
 assignment, we can easily make a longer period on the timer. The timer
 period dictates the minimum stuck assignment time -- I can update the
 description with a clarification.
 
  kturner wrote:
  I was thinking that once an assignment is considered stuck, that
 each time the timer kicks a check (I think its either 5 secs or 1 sec, not
 sure) that it will cause a spam.  Was thinking this could be increased to
 produce less spam.  The period of the timer could be a function of
 tserver.assignment.duration.warning, like 1/4 or 1/2.

 bq. The period of the timer could be a function of
 tserver.assignment.duration.warning, like 1/4 or 1/2.

 That would work, unless the user changed the value of the duration
 warning. It would still fire at the old period (unless I'm much trickier
 about scheduling the task to run).

 Regardless need to think some more about preventing spam.


 - Josh


 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/27654/#review60185
 ---


 On Nov. 6, 2014, 12:58 a.m., Josh Elser wrote:
 
  ---
  This is an automatically generated e-mail. To reply, visit:
  https://reviews.apache.org/r/27654/
  ---
 
  (Updated Nov. 6, 2014, 12:58 a.m.)
 
 
  Review request for accumulo.
 
 
  Bugs: ACCUMULO-3304
  https://issues.apache.org/jira/browse/ACCUMULO-3304
 
 
  Repository: accumulo
 
 
  Description
  ---
 
  Watches assignments and reports when an assignment is running for
 longer than a configured time.
 
 
  Diffs
  -
 
core/src/main/java/org/apache/accumulo/core/conf/Property.java 56f3d9c
 
  
 server/tserver/src/main/java/org/apache/accumulo/tserver/ActiveAssignmentRunnable.java
 PRE-CREATION
 
  
 server/tserver/src/main/java/org/apache/accumulo/tserver/RunnableStartedAt.java
 PRE-CREATION
 
  server/tserver/src/main/java/org/apache/accumulo/tserver/TabletServer.java
 94be0bb
 
  
 server/tserver/src/main/java/org/apache/accumulo/tserver/TabletServerResourceManager.java
 935ffeb
 
  Diff: https://reviews.apache.org/r/27654/diff/
 
 
  Testing
  ---
 
  Very minimal.
 
 
  Thanks,
 
  Josh Elser
 
 

Re: Review Request 26943: Changes to ZK for implicit retry handling

[Eric Newton]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26943/#review57379
---



fate/src/main/java/org/apache/accumulo/fate/zookeeper/ZooCache.java
https://reviews.apache.org/r/26943/#comment98067

nit: spelling wil



fate/src/main/java/org/apache/accumulo/fate/zookeeper/ZooUtil.java
https://reviews.apache.org/r/26943/#comment98092

maybe add hashCode and equals?



fate/src/main/java/org/apache/accumulo/fate/zookeeper/ZooUtil.java
https://reviews.apache.org/r/26943/#comment98093

nit: braces


- Eric Newton


On Oct. 20, 2014, 4:28 p.m., Josh Elser wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/26943/
 ---
 
 (Updated Oct. 20, 2014, 4:28 p.m.)
 
 
 Review request for accumulo.
 
 
 Bugs: ACCUMULO-3242
 https://issues.apache.org/jira/browse/ACCUMULO-3242
 
 
 Repository: accumulo
 
 
 Description
 ---
 
 We didn't do some things correctly WRT zookeeper retries. Attempt to make it 
 better. Made a review in case people want to give some feedback
 
 
 Diffs
 -
 
   fate/src/main/java/org/apache/accumulo/fate/zookeeper/IZooReaderWriter.java 
 afc2250 
   fate/src/main/java/org/apache/accumulo/fate/zookeeper/Retry.java 4a37172 
   fate/src/main/java/org/apache/accumulo/fate/zookeeper/RetryFactory.java 
 3fcb738 
   fate/src/main/java/org/apache/accumulo/fate/zookeeper/ZooCache.java d9eb243 
   fate/src/main/java/org/apache/accumulo/fate/zookeeper/ZooReader.java 
 994f395 
   fate/src/main/java/org/apache/accumulo/fate/zookeeper/ZooReaderWriter.java 
 b29b88a 
   fate/src/main/java/org/apache/accumulo/fate/zookeeper/ZooUtil.java e0d8831 
   fate/src/test/java/org/apache/accumulo/fate/zookeeper/RetryTest.java 
 PRE-CREATION 
   
 fate/src/test/java/org/apache/accumulo/fate/zookeeper/ZooReaderWriterTest.java
  PRE-CREATION 
   server/base/src/main/java/org/apache/accumulo/server/init/Initialize.java 
 5cbffc3 
   
 server/base/src/main/java/org/apache/accumulo/server/security/handler/ZKPermHandler.java
  3cac2e9 
 
 Diff: https://reviews.apache.org/r/26943/diff/
 
 
 Testing
 ---
 
 UTs/ITs
 
 
 Thanks,
 
 Josh Elser
 

Re: Review Request 26507: ACCUMULO-3177 Create a per table volume chooser and ACCUMULO-3178 Create example preferred volumes chooser

[Eric Newton]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26507/#review56558
---

Ship it!


Ship It!

- Eric Newton


On Oct. 14, 2014, 3:51 p.m., Jenna Huston wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/26507/
 ---
 
 (Updated Oct. 14, 2014, 3:51 p.m.)
 
 
 Review request for accumulo.
 
 
 Bugs: ACCUMULO-3177 and ACCUMULO-3178
 https://issues.apache.org/jira/browse/ACCUMULO-3177
 https://issues.apache.org/jira/browse/ACCUMULO-3178
 
 
 Repository: accumulo
 
 
 Description
 ---
 
 Added a per table volume chooser that allows tables to be given a specific 
 volume chooser.  The second patch, ACCUMULO-3178, adds an example, a 
 preferred volume chooser which gives the preferred volume for a table.  When 
 a table chooser is not specified, or a preferred volume is not specified 
 then, the default chooser is the RandomVolumeChooser.
 
 
 Diffs
 -
 
   core/src/main/java/org/apache/accumulo/core/conf/Property.java ad4fe92 
   
 server/base/src/main/java/org/apache/accumulo/server/fs/PerTableVolumeChooser.java
  PRE-CREATION 
   
 server/base/src/main/java/org/apache/accumulo/server/fs/RandomVolumeChooser.java
  2760b07 
   server/base/src/main/java/org/apache/accumulo/server/fs/VolumeChooser.java 
 8713c97 
   
 server/base/src/main/java/org/apache/accumulo/server/fs/VolumeChooserEnvironment.java
  PRE-CREATION 
   server/base/src/main/java/org/apache/accumulo/server/fs/VolumeManager.java 
 cbfdb5e 
   
 server/base/src/main/java/org/apache/accumulo/server/fs/VolumeManagerImpl.java
  b2341b6 
   server/base/src/main/java/org/apache/accumulo/server/fs/VolumeUtil.java 
 82b77ee 
   server/base/src/main/java/org/apache/accumulo/server/init/Initialize.java 
 4305e71 
   server/base/src/main/java/org/apache/accumulo/server/util/FileUtil.java 
 0f7ac22 
   
 server/base/src/main/java/org/apache/accumulo/server/util/MetadataTableUtil.java
  8d14cad 
   
 server/base/src/main/java/org/apache/accumulo/server/util/TabletOperations.java
  b8e7113 
   server/master/src/main/java/org/apache/accumulo/master/Master.java b435b0f 
   
 server/master/src/main/java/org/apache/accumulo/master/TabletGroupWatcher.java
  0a3d1d0 
   
 server/master/src/main/java/org/apache/accumulo/master/tableOps/CreateTable.java
  f56c1e2 
   server/tserver/src/main/java/org/apache/accumulo/tserver/log/DfsLogger.java 
 e923ebc 
   
 server/tserver/src/test/java/org/apache/accumulo/tserver/TabletServerSyncCheckTest.java
  dad9a75 
 
 Diff: https://reviews.apache.org/r/26507/diff/
 
 
 Testing
 ---
 
 New IT in the patch for ACCUMULO-3178.  Could not test ACCUMULO-3177 without 
 an example chooser.
 
 
 File Attachments
 
 
 Diff for 3178
   
 https://reviews.apache.org/media/uploaded/files/2014/10/09/07d2693e-9acc-438b-9b13-667bde467590__0001-ACCUMULO-3178-Create-example-preferred-volumes-choos.patch
 Updated Diff for 3178
   
 https://reviews.apache.org/media/uploaded/files/2014/10/14/2b5aa6b9-92d0-4938-bcf9-2021cd0212ee__0001-ACCUMULO-3178-Create-example-preferred-volumes-choos.patch
 
 
 Thanks,
 
 Jenna Huston
 

Re: Review Request 26572: ACCUMULO-898

[Eric Newton]

/main/java/org/apache/accumulo/tserver/tablet/DatafileManager.java
 78a2ed6 
  
server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/MinorCompactionTask.java
 5e5f31f 
  server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/Tablet.java 
0194778 
  shell/src/main/java/org/apache/accumulo/shell/Shell.java 16742b8 
  shell/src/main/java/org/apache/accumulo/shell/commands/TraceCommand.java 
7f63570 
  start/pom.xml 61e5b32 
  test/pom.xml 78909e1 
  test/src/main/java/org/apache/accumulo/test/GetMasterStats.java 0ba4ab2 
  test/src/main/java/org/apache/accumulo/test/TestIngest.java 0548f4c 
  test/src/main/java/org/apache/accumulo/test/VerifyIngest.java 74b03e4 
  test/src/main/java/org/apache/accumulo/test/WrongTabletTest.java b563ed9 
  test/src/main/java/org/apache/accumulo/test/continuous/ContinuousIngest.java 
d6a16df 
  
test/src/main/java/org/apache/accumulo/test/continuous/ContinuousStatsCollector.java
 b36dbeb 
  test/src/main/java/org/apache/accumulo/test/continuous/ContinuousWalk.java 
262f7b0 
  test/src/main/java/org/apache/accumulo/test/functional/ZombieTServer.java 
3b1aeaf 
  
test/src/main/java/org/apache/accumulo/test/randomwalk/concurrent/Shutdown.java 
ad93c78 
  
test/src/main/java/org/apache/accumulo/test/randomwalk/concurrent/StartAll.java 
4fc94d8 
  test/src/test/java/org/apache/accumulo/test/ConditionalWriterIT.java 570a53c 
  test/src/test/java/org/apache/accumulo/test/ShellServerIT.java 5a068af 
  test/src/test/java/org/apache/accumulo/test/VolumeIT.java 5e54957 
  test/src/test/java/org/apache/accumulo/test/functional/AbstractMacIT.java 
644b05e 
  
test/src/test/java/org/apache/accumulo/test/functional/BalanceAfterCommsFailureIT.java
 7b4a774 
  test/src/test/java/org/apache/accumulo/test/functional/ConfigurableMacIT.java 
eab14f5 
  
test/src/test/java/org/apache/accumulo/test/functional/DynamicThreadPoolsIT.java
 87497b9 
  test/src/test/java/org/apache/accumulo/test/functional/ExamplesIT.java 
210e057 
  test/src/test/java/org/apache/accumulo/test/functional/MetadataMaxFiles.java 
6b8d9b3 
  
test/src/test/java/org/apache/accumulo/test/functional/SimpleBalancerFairnessIT.java
 966c150 
  
test/src/test/java/org/apache/accumulo/test/replication/CyclicReplicationIT.java
 93c8650 
  trace/pom.xml aacfb56 
  trace/src/main/java/org/apache/accumulo/trace/instrument/CountSampler.java 
9a5bdbb 
  trace/src/main/java/org/apache/accumulo/trace/instrument/Sampler.java 4abb40a 
  trace/src/main/java/org/apache/accumulo/trace/instrument/Span.java 5267174 
  
trace/src/main/java/org/apache/accumulo/trace/instrument/SpanReceiverHost.java 
PRE-CREATION 
  trace/src/main/java/org/apache/accumulo/trace/instrument/Trace.java 19171c4 
  trace/src/main/java/org/apache/accumulo/trace/instrument/TraceCallable.java 
c3072b1 
  
trace/src/main/java/org/apache/accumulo/trace/instrument/TraceExecutorService.java
 04dcc39 
  trace/src/main/java/org/apache/accumulo/trace/instrument/TraceProxy.java 
cb93210 
  trace/src/main/java/org/apache/accumulo/trace/instrument/TraceRunnable.java 
41c765d 
  trace/src/main/java/org/apache/accumulo/trace/instrument/TraceUtil.java 
PRE-CREATION 
  trace/src/main/java/org/apache/accumulo/trace/instrument/Tracer.java d70aeea 
  trace/src/main/java/org/apache/accumulo/trace/instrument/impl/MilliSpan.java 
b641a2c 
  trace/src/main/java/org/apache/accumulo/trace/instrument/impl/NullSpan.java 
916b6cf 
  
trace/src/main/java/org/apache/accumulo/trace/instrument/impl/RootMilliSpan.java
 c25e644 
  
trace/src/main/java/org/apache/accumulo/trace/instrument/receivers/AsyncSpanReceiver.java
 4eebd69 
  
trace/src/main/java/org/apache/accumulo/trace/instrument/receivers/LogSpans.java
 dfed660 
  
trace/src/main/java/org/apache/accumulo/trace/instrument/receivers/SendSpansViaThrift.java
 4967d97 
  
trace/src/main/java/org/apache/accumulo/trace/instrument/receivers/SpanReceiver.java
 b44e51e 
  
trace/src/main/java/org/apache/accumulo/trace/instrument/receivers/ZooSpanClient.java
 0ed92fc 
  
trace/src/main/java/org/apache/accumulo/trace/instrument/receivers/ZooTraceClient.java
 PRE-CREATION 
  
trace/src/main/java/org/apache/accumulo/trace/instrument/thrift/RpcClientInvocationHandler.java
 ea57fe7 
  
trace/src/main/java/org/apache/accumulo/trace/instrument/thrift/RpcServerInvocationHandler.java
 4188bf4 
  trace/src/main/java/org/apache/accumulo/trace/thrift/Annotation.java 
PRE-CREATION 
  trace/src/main/java/org/apache/accumulo/trace/thrift/RemoteSpan.java 416ae17 
  trace/src/main/thrift/trace.thrift 76bcafe 
  
trace/src/test/java/org/apache/accumulo/trace/instrument/CountSamplerTest.java 
7b5e970 
  trace/src/test/java/org/apache/accumulo/trace/instrument/PerformanceTest.java 
2e5aaa5 
  trace/src/test/java/org/apache/accumulo/trace/instrument/TracerTest.java 
f338bd8 

Diff: https://reviews.apache.org/r/26572/diff/


Testing
---


Thanks,

Eric Newton

Review Request 26572: ACCUMULO-898

[Eric Newton]

/tablet/DatafileManager.java
 78a2ed6 
  
server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/MinorCompactionTask.java
 5e5f31f 
  server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/Tablet.java 
0194778 
  shell/src/main/java/org/apache/accumulo/shell/Shell.java 16742b8 
  shell/src/main/java/org/apache/accumulo/shell/commands/TraceCommand.java 
7f63570 
  start/pom.xml 61e5b32 
  test/pom.xml 78909e1 
  test/src/main/java/org/apache/accumulo/test/GetMasterStats.java 0ba4ab2 
  test/src/main/java/org/apache/accumulo/test/TestIngest.java 0548f4c 
  test/src/main/java/org/apache/accumulo/test/VerifyIngest.java 74b03e4 
  test/src/main/java/org/apache/accumulo/test/WrongTabletTest.java b563ed9 
  test/src/main/java/org/apache/accumulo/test/continuous/ContinuousIngest.java 
d6a16df 
  
test/src/main/java/org/apache/accumulo/test/continuous/ContinuousStatsCollector.java
 b36dbeb 
  test/src/main/java/org/apache/accumulo/test/continuous/ContinuousWalk.java 
262f7b0 
  test/src/main/java/org/apache/accumulo/test/functional/ZombieTServer.java 
3b1aeaf 
  
test/src/main/java/org/apache/accumulo/test/randomwalk/concurrent/Shutdown.java 
ad93c78 
  
test/src/main/java/org/apache/accumulo/test/randomwalk/concurrent/StartAll.java 
4fc94d8 
  test/src/test/java/org/apache/accumulo/test/ConditionalWriterIT.java 570a53c 
  test/src/test/java/org/apache/accumulo/test/ShellServerIT.java 5a068af 
  test/src/test/java/org/apache/accumulo/test/VolumeIT.java 5e54957 
  test/src/test/java/org/apache/accumulo/test/functional/AbstractMacIT.java 
644b05e 
  
test/src/test/java/org/apache/accumulo/test/functional/BalanceAfterCommsFailureIT.java
 7b4a774 
  test/src/test/java/org/apache/accumulo/test/functional/ConfigurableMacIT.java 
eab14f5 
  
test/src/test/java/org/apache/accumulo/test/functional/DynamicThreadPoolsIT.java
 87497b9 
  test/src/test/java/org/apache/accumulo/test/functional/ExamplesIT.java 
210e057 
  test/src/test/java/org/apache/accumulo/test/functional/MetadataMaxFiles.java 
6b8d9b3 
  
test/src/test/java/org/apache/accumulo/test/functional/SimpleBalancerFairnessIT.java
 966c150 
  
test/src/test/java/org/apache/accumulo/test/replication/CyclicReplicationIT.java
 93c8650 
  trace/pom.xml aacfb56 
  trace/src/main/java/org/apache/accumulo/trace/instrument/CountSampler.java 
9a5bdbb 
  trace/src/main/java/org/apache/accumulo/trace/instrument/Sampler.java 4abb40a 
  trace/src/main/java/org/apache/accumulo/trace/instrument/Span.java 5267174 
  
trace/src/main/java/org/apache/accumulo/trace/instrument/SpanReceiverHost.java 
PRE-CREATION 
  trace/src/main/java/org/apache/accumulo/trace/instrument/Trace.java 19171c4 
  trace/src/main/java/org/apache/accumulo/trace/instrument/TraceCallable.java 
c3072b1 
  
trace/src/main/java/org/apache/accumulo/trace/instrument/TraceExecutorService.java
 04dcc39 
  trace/src/main/java/org/apache/accumulo/trace/instrument/TraceProxy.java 
cb93210 
  trace/src/main/java/org/apache/accumulo/trace/instrument/TraceRunnable.java 
41c765d 
  trace/src/main/java/org/apache/accumulo/trace/instrument/TraceUtil.java 
PRE-CREATION 
  trace/src/main/java/org/apache/accumulo/trace/instrument/Tracer.java d70aeea 
  trace/src/main/java/org/apache/accumulo/trace/instrument/impl/MilliSpan.java 
b641a2c 
  trace/src/main/java/org/apache/accumulo/trace/instrument/impl/NullSpan.java 
916b6cf 
  
trace/src/main/java/org/apache/accumulo/trace/instrument/impl/RootMilliSpan.java
 c25e644 
  
trace/src/main/java/org/apache/accumulo/trace/instrument/receivers/AsyncSpanReceiver.java
 4eebd69 
  
trace/src/main/java/org/apache/accumulo/trace/instrument/receivers/LogSpans.java
 dfed660 
  
trace/src/main/java/org/apache/accumulo/trace/instrument/receivers/SendSpansViaThrift.java
 4967d97 
  
trace/src/main/java/org/apache/accumulo/trace/instrument/receivers/SpanReceiver.java
 b44e51e 
  
trace/src/main/java/org/apache/accumulo/trace/instrument/receivers/ZooSpanClient.java
 0ed92fc 
  
trace/src/main/java/org/apache/accumulo/trace/instrument/receivers/ZooTraceClient.java
 PRE-CREATION 
  
trace/src/main/java/org/apache/accumulo/trace/instrument/thrift/RpcClientInvocationHandler.java
 ea57fe7 
  
trace/src/main/java/org/apache/accumulo/trace/instrument/thrift/RpcServerInvocationHandler.java
 4188bf4 
  trace/src/main/java/org/apache/accumulo/trace/thrift/Annotation.java 
PRE-CREATION 
  trace/src/main/java/org/apache/accumulo/trace/thrift/RemoteSpan.java 416ae17 
  trace/src/main/thrift/trace.thrift 76bcafe 
  
trace/src/test/java/org/apache/accumulo/trace/instrument/CountSamplerTest.java 
7b5e970 
  trace/src/test/java/org/apache/accumulo/trace/instrument/PerformanceTest.java 
2e5aaa5 
  trace/src/test/java/org/apache/accumulo/trace/instrument/TracerTest.java 
f338bd8 

Diff: https://reviews.apache.org/r/26572/diff/


Testing
---


Thanks,

Eric Newton

Re: Review Request 26572: ACCUMULO-898

[Eric Newton]

/main/java/org/apache/accumulo/tserver/tablet/DatafileManager.java
 78a2ed6 
  
server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/MinorCompactionTask.java
 5e5f31f 
  server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/Tablet.java 
0194778 
  shell/src/main/java/org/apache/accumulo/shell/Shell.java 16742b8 
  shell/src/main/java/org/apache/accumulo/shell/commands/TraceCommand.java 
7f63570 
  start/pom.xml 61e5b32 
  test/pom.xml 78909e1 
  test/src/main/java/org/apache/accumulo/test/GetMasterStats.java 0ba4ab2 
  test/src/main/java/org/apache/accumulo/test/TestIngest.java 0548f4c 
  test/src/main/java/org/apache/accumulo/test/VerifyIngest.java 74b03e4 
  test/src/main/java/org/apache/accumulo/test/WrongTabletTest.java b563ed9 
  test/src/main/java/org/apache/accumulo/test/continuous/ContinuousIngest.java 
d6a16df 
  
test/src/main/java/org/apache/accumulo/test/continuous/ContinuousStatsCollector.java
 b36dbeb 
  test/src/main/java/org/apache/accumulo/test/continuous/ContinuousWalk.java 
262f7b0 
  test/src/main/java/org/apache/accumulo/test/functional/ZombieTServer.java 
3b1aeaf 
  
test/src/main/java/org/apache/accumulo/test/randomwalk/concurrent/Shutdown.java 
ad93c78 
  
test/src/main/java/org/apache/accumulo/test/randomwalk/concurrent/StartAll.java 
4fc94d8 
  test/src/test/java/org/apache/accumulo/test/ConditionalWriterIT.java 570a53c 
  test/src/test/java/org/apache/accumulo/test/ShellServerIT.java 5a068af 
  test/src/test/java/org/apache/accumulo/test/VolumeIT.java 5e54957 
  test/src/test/java/org/apache/accumulo/test/functional/AbstractMacIT.java 
644b05e 
  
test/src/test/java/org/apache/accumulo/test/functional/BalanceAfterCommsFailureIT.java
 7b4a774 
  test/src/test/java/org/apache/accumulo/test/functional/ConfigurableMacIT.java 
eab14f5 
  
test/src/test/java/org/apache/accumulo/test/functional/DynamicThreadPoolsIT.java
 87497b9 
  test/src/test/java/org/apache/accumulo/test/functional/ExamplesIT.java 
210e057 
  test/src/test/java/org/apache/accumulo/test/functional/MetadataMaxFiles.java 
6b8d9b3 
  
test/src/test/java/org/apache/accumulo/test/functional/SimpleBalancerFairnessIT.java
 966c150 
  
test/src/test/java/org/apache/accumulo/test/replication/CyclicReplicationIT.java
 93c8650 
  trace/pom.xml aacfb56 
  trace/src/main/java/org/apache/accumulo/trace/instrument/CountSampler.java 
9a5bdbb 
  trace/src/main/java/org/apache/accumulo/trace/instrument/Sampler.java 4abb40a 
  trace/src/main/java/org/apache/accumulo/trace/instrument/Span.java 5267174 
  
trace/src/main/java/org/apache/accumulo/trace/instrument/SpanReceiverHost.java 
PRE-CREATION 
  trace/src/main/java/org/apache/accumulo/trace/instrument/Trace.java 19171c4 
  trace/src/main/java/org/apache/accumulo/trace/instrument/TraceCallable.java 
c3072b1 
  
trace/src/main/java/org/apache/accumulo/trace/instrument/TraceExecutorService.java
 04dcc39 
  trace/src/main/java/org/apache/accumulo/trace/instrument/TraceProxy.java 
cb93210 
  trace/src/main/java/org/apache/accumulo/trace/instrument/TraceRunnable.java 
41c765d 
  trace/src/main/java/org/apache/accumulo/trace/instrument/TraceUtil.java 
PRE-CREATION 
  trace/src/main/java/org/apache/accumulo/trace/instrument/Tracer.java d70aeea 
  trace/src/main/java/org/apache/accumulo/trace/instrument/impl/MilliSpan.java 
b641a2c 
  trace/src/main/java/org/apache/accumulo/trace/instrument/impl/NullSpan.java 
916b6cf 
  
trace/src/main/java/org/apache/accumulo/trace/instrument/impl/RootMilliSpan.java
 c25e644 
  
trace/src/main/java/org/apache/accumulo/trace/instrument/receivers/AsyncSpanReceiver.java
 4eebd69 
  
trace/src/main/java/org/apache/accumulo/trace/instrument/receivers/LogSpans.java
 dfed660 
  
trace/src/main/java/org/apache/accumulo/trace/instrument/receivers/SendSpansViaThrift.java
 4967d97 
  
trace/src/main/java/org/apache/accumulo/trace/instrument/receivers/SpanReceiver.java
 b44e51e 
  
trace/src/main/java/org/apache/accumulo/trace/instrument/receivers/ZooSpanClient.java
 0ed92fc 
  
trace/src/main/java/org/apache/accumulo/trace/instrument/receivers/ZooTraceClient.java
 PRE-CREATION 
  
trace/src/main/java/org/apache/accumulo/trace/instrument/thrift/RpcClientInvocationHandler.java
 ea57fe7 
  
trace/src/main/java/org/apache/accumulo/trace/instrument/thrift/RpcServerInvocationHandler.java
 4188bf4 
  trace/src/main/java/org/apache/accumulo/trace/thrift/Annotation.java 
PRE-CREATION 
  trace/src/main/java/org/apache/accumulo/trace/thrift/RemoteSpan.java 416ae17 
  trace/src/main/thrift/trace.thrift 76bcafe 
  
trace/src/test/java/org/apache/accumulo/trace/instrument/CountSamplerTest.java 
7b5e970 
  trace/src/test/java/org/apache/accumulo/trace/instrument/PerformanceTest.java 
2e5aaa5 
  trace/src/test/java/org/apache/accumulo/trace/instrument/TracerTest.java 
f338bd8 

Diff: https://reviews.apache.org/r/26572/diff/


Testing
---


Thanks,

Eric Newton

Re: Review Request 26572: ACCUMULO-898

[Eric Newton]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26572/#review56204
---


DistributedTrace.enable, Trace.on, Trace.off should be re-created and 
deprecated to help users transition away from documented APIs. Is there some 
reason TraceUtil isn't named Tracer?  Seems like it would save a lot of the 
machanical changes.


core/src/main/java/org/apache/accumulo/core/cli/ClientOpts.java
https://reviews.apache.org/r/26572/#comment96520

Why not call TraceUtil Trace?  Seems like it would eliminate most of the 
mechanical changes.



core/src/main/java/org/apache/accumulo/core/client/impl/TabletServerBatchWriter.java
https://reviews.apache.org/r/26572/#comment96521

Span could be a trivial sub-class of TraceScope, and maintain compatibility.



core/src/main/java/org/apache/accumulo/core/client/impl/TabletServerBatchWriter.java
https://reviews.apache.org/r/26572/#comment96522

I think there's still a way to maintain some compatibility, though I'm not 
concerned about maintaining it inside accumulo.



core/src/main/java/org/apache/accumulo/core/util/ThriftUtil.java
https://reviews.apache.org/r/26572/#comment96523

nit: whitespace?



minicluster/src/main/java/org/apache/accumulo/minicluster/impl/MiniAccumuloClusterImpl.java
https://reviews.apache.org/r/26572/#comment96524

Why add the IPv4Stack change?  Is this related to tracing?



minicluster/src/main/java/org/apache/accumulo/minicluster/impl/MiniAccumuloClusterImpl.java
https://reviews.apache.org/r/26572/#comment96525

What's the reason for this change?



server/monitor/src/main/java/org/apache/accumulo/monitor/Monitor.java
https://reviews.apache.org/r/26572/#comment96526

Why move this? It is probably the right change, but it could probably go in 
a separate ticket.



server/monitor/src/main/java/org/apache/accumulo/monitor/servlets/trace/ShowTrace.java
https://reviews.apache.org/r/26572/#comment96527

This expression is now long enough to push into a local method.



server/monitor/src/test/java/org/apache/accumulo/monitor/ShowTraceLinkTypeTest.java
https://reviews.apache.org/r/26572/#comment96528

Hoist Collections.ByteBuffer, ByteBufferemptyMap() into a constant for 
readability?



server/tserver/src/main/java/org/apache/accumulo/tserver/replication/AccumuloReplicaSystem.java
https://reviews.apache.org/r/26572/#comment96530

Why not make a version of TraceUtil.data that takes a TraceScope?



test/src/test/java/org/apache/accumulo/test/ConditionalWriterIT.java
https://reviews.apache.org/r/26572/#comment96532

Is this related to tracing changes?


- Eric Newton


On Oct. 10, 2014, 7:12 p.m., Eric Newton wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/26572/
 ---
 
 (Updated Oct. 10, 2014, 7:12 p.m.)
 
 
 Review request for accumulo.
 
 
 Repository: accumulo
 
 
 Description
 ---
 
 Switch to HTrace
 
 
 Diffs
 -
 
   core/pom.xml 086ebc0 
   core/src/main/java/org/apache/accumulo/core/cli/ClientOpts.java e582160 
   
 core/src/main/java/org/apache/accumulo/core/client/impl/ConditionalWriterImpl.java
  e8af187 
   core/src/main/java/org/apache/accumulo/core/client/impl/ConnectorImpl.java 
 62ab6a4 
   
 core/src/main/java/org/apache/accumulo/core/client/impl/InstanceOperationsImpl.java
  c5f7634 
   
 core/src/main/java/org/apache/accumulo/core/client/impl/NamespaceOperationsImpl.java
  7087ac5 
   
 core/src/main/java/org/apache/accumulo/core/client/impl/ReplicationOperationsImpl.java
  f820aa4 
   
 core/src/main/java/org/apache/accumulo/core/client/impl/SecurityOperationsImpl.java
  e9c057e 
   
 core/src/main/java/org/apache/accumulo/core/client/impl/TableOperationsImpl.java
  e46b9c9 
   
 core/src/main/java/org/apache/accumulo/core/client/impl/TabletServerBatchReaderIterator.java
  d2ca60e 
   
 core/src/main/java/org/apache/accumulo/core/client/impl/TabletServerBatchWriter.java
  5eec397 
   core/src/main/java/org/apache/accumulo/core/client/impl/ThriftScanner.java 
 c49330e 
   core/src/main/java/org/apache/accumulo/core/client/impl/Writer.java 0358a88 
   core/src/main/java/org/apache/accumulo/core/trace/DistributedTrace.java 
 83f5c26 
   core/src/main/java/org/apache/accumulo/core/trace/SpanTree.java 772a133 
   core/src/main/java/org/apache/accumulo/core/trace/TraceDump.java b44cc3e 
   core/src/main/java/org/apache/accumulo/core/trace/TraceFormatter.java 
 9d860d9 
   core/src/main/java/org/apache/accumulo/core/trace/ZooTraceClient.java 
 4c8b837 
   core/src/main/java/org/apache/accumulo/core/util/ThriftUtil.java da4e567 
   
 examples/simple/src/main/java/org/apache/accumulo/examples/simple/client/TracingExample.java
  a542263 
   
 minicluster/src/main/java/org

Re: Review Request 26572: ACCUMULO-898

[Eric Newton]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26572/#review56209
---


Missed updating the instructions in 
docs/src/main/resources/distributedTracing.html

- Eric Newton


On Oct. 10, 2014, 7:12 p.m., Eric Newton wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/26572/
 ---
 
 (Updated Oct. 10, 2014, 7:12 p.m.)
 
 
 Review request for accumulo.
 
 
 Repository: accumulo
 
 
 Description
 ---
 
 Switch to HTrace
 
 
 Diffs
 -
 
   core/pom.xml 086ebc0 
   core/src/main/java/org/apache/accumulo/core/cli/ClientOpts.java e582160 
   
 core/src/main/java/org/apache/accumulo/core/client/impl/ConditionalWriterImpl.java
  e8af187 
   core/src/main/java/org/apache/accumulo/core/client/impl/ConnectorImpl.java 
 62ab6a4 
   
 core/src/main/java/org/apache/accumulo/core/client/impl/InstanceOperationsImpl.java
  c5f7634 
   
 core/src/main/java/org/apache/accumulo/core/client/impl/NamespaceOperationsImpl.java
  7087ac5 
   
 core/src/main/java/org/apache/accumulo/core/client/impl/ReplicationOperationsImpl.java
  f820aa4 
   
 core/src/main/java/org/apache/accumulo/core/client/impl/SecurityOperationsImpl.java
  e9c057e 
   
 core/src/main/java/org/apache/accumulo/core/client/impl/TableOperationsImpl.java
  e46b9c9 
   
 core/src/main/java/org/apache/accumulo/core/client/impl/TabletServerBatchReaderIterator.java
  d2ca60e 
   
 core/src/main/java/org/apache/accumulo/core/client/impl/TabletServerBatchWriter.java
  5eec397 
   core/src/main/java/org/apache/accumulo/core/client/impl/ThriftScanner.java 
 c49330e 
   core/src/main/java/org/apache/accumulo/core/client/impl/Writer.java 0358a88 
   core/src/main/java/org/apache/accumulo/core/trace/DistributedTrace.java 
 83f5c26 
   core/src/main/java/org/apache/accumulo/core/trace/SpanTree.java 772a133 
   core/src/main/java/org/apache/accumulo/core/trace/TraceDump.java b44cc3e 
   core/src/main/java/org/apache/accumulo/core/trace/TraceFormatter.java 
 9d860d9 
   core/src/main/java/org/apache/accumulo/core/trace/ZooTraceClient.java 
 4c8b837 
   core/src/main/java/org/apache/accumulo/core/util/ThriftUtil.java da4e567 
   
 examples/simple/src/main/java/org/apache/accumulo/examples/simple/client/TracingExample.java
  a542263 
   
 minicluster/src/main/java/org/apache/accumulo/minicluster/impl/MiniAccumuloClusterImpl.java
  1fb5901 
   pom.xml ebc2f2f 
   server/base/pom.xml 60762be 
   server/base/src/main/java/org/apache/accumulo/server/Accumulo.java 7bb3d71 
   
 server/base/src/main/java/org/apache/accumulo/server/client/BulkImporter.java 
 7097079 
   
 server/base/src/main/java/org/apache/accumulo/server/master/LiveTServerSet.java
  3dbc6ba 
   
 server/base/src/main/java/org/apache/accumulo/server/master/balancer/TabletBalancer.java
  59d3e0b 
   
 server/base/src/main/java/org/apache/accumulo/server/trace/TraceFSDataInputStream.java
  5162e01 
   
 server/base/src/main/java/org/apache/accumulo/server/trace/TraceFileSystem.java
  d3fbad7 
   server/base/src/main/java/org/apache/accumulo/server/util/Admin.java 
 d85d61e 
   
 server/base/src/main/java/org/apache/accumulo/server/util/VerifyTabletAssignments.java
  0fbeb6b 
   
 server/gc/src/main/java/org/apache/accumulo/gc/GarbageCollectWriteAheadLogs.java
  9646be9 
   
 server/gc/src/main/java/org/apache/accumulo/gc/GarbageCollectionAlgorithm.java
  0f5cada 
   server/gc/src/main/java/org/apache/accumulo/gc/SimpleGarbageCollector.java 
 10fd7f5 
   
 server/gc/src/main/java/org/apache/accumulo/gc/replication/CloseWriteAheadLogReferences.java
  74da72d 
   server/master/src/main/java/org/apache/accumulo/master/Master.java b435b0f 
   
 server/master/src/main/java/org/apache/accumulo/master/metrics/ReplicationMetrics.java
  06a653d 
   
 server/master/src/main/java/org/apache/accumulo/master/replication/ReplicationDriver.java
  a52f743 
   
 server/master/src/main/java/org/apache/accumulo/master/replication/StatusMaker.java
  68dd005 
   
 server/master/src/main/java/org/apache/accumulo/master/replication/WorkMaker.java
  da17bba 
   
 server/master/src/main/java/org/apache/accumulo/master/tableOps/BulkImport.java
  b55b315 
   
 server/master/src/main/java/org/apache/accumulo/master/tableOps/TraceRepo.java
  9b1a735 
   server/monitor/src/main/java/org/apache/accumulo/monitor/Monitor.java 
 7a724f8 
   
 server/monitor/src/main/java/org/apache/accumulo/monitor/servlets/TServersServlet.java
  452f790 
   
 server/monitor/src/main/java/org/apache/accumulo/monitor/servlets/trace/ShowTrace.java
  a476201 
   
 server/monitor/src/test/java/org/apache/accumulo/monitor/ShowTraceLinkTypeTest.java
  a630434 
   server/tracer/src/main/java/org/apache/accumulo/tracer/TraceServer.java 
 189bb39

Re: [VOTE] Apache Accumulo 1.6.1 RC1

[Eric Newton]

 (which needs to be signed)


It is signed... I forgot I have to add trust:

$ gpg --update-trustdb


Thanks Corey!

On Tue, Sep 23, 2014 at 9:19 AM, Eric Newton eric.new...@gmail.com wrote:

 +1
 Verified signature (which needs to be signed)
 Verified ingest performance (on a single node)
 Looked over the user's manual from the binary tarball
 Ran all unit and integration tests


 On Fri, Sep 19, 2014 at 10:49 PM, Corey Nolet cjno...@gmail.com wrote:

 Devs,

 Please consider the following candidate for Apache Accumulo 1.6.1

 Branch: 1.6.1-rc1
 SHA1: 88c5473b3b49d797d3dabebd12fe517e9b248ba2
 Staging Repository:
 *
 https://repository.apache.org/content/repositories/orgapacheaccumulo-1017/
 
 https://repository.apache.org/content/repositories/orgapacheaccumulo-1017/
 *

 Source tarball:
 *
 http://repository.apache.org/content/repositories/orgapacheaccumulo-1017/org/apache/accumulo/accumulo/1.6.1/accumulo-1.6.1-src.tar.gz
 
 http://repository.apache.org/content/repositories/orgapacheaccumulo-1017/org/apache/accumulo/accumulo/1.6.1/accumulo-1.6.1-src.tar.gz
 *
 Binary tarball:
 *
 http://repository.apache.org/content/repositories/orgapacheaccumulo-1017/org/apache/accumulo/accumulo/1.6.1/accumulo-1.6.1-bin.tar.gz
 
 http://repository.apache.org/content/repositories/orgapacheaccumulo-1017/org/apache/accumulo/accumulo/1.6.1/accumulo-1.6.1-bin.tar.gz
 *
 (Append .sha1, .md5 or .asc to download the signature/hash for a
 given artifact.)

 Signing keys available at: https://www.apache.org/dist/accumulo/KEYS

 Over 1.6.1, we have 188 issues resolved
 *
 https://git-wip-us.apache.org/repos/asf?p=accumulo.git;a=blob;f=CHANGES;h=91b9d31e3b9dc53f1a576cc49bbc061919eb0070;hb=1.6.1-rc1
 
 https://git-wip-us.apache.org/repos/asf?p=accumulo.git;a=blob;f=CHANGES;h=91b9d31e3b9dc53f1a576cc49bbc061919eb0070;hb=1.6.1-rc1
 *

 Testing: All unit and functional tests are passing.

 Vote will be open until Thursday, September 25th 12:00AM UTC (9/24 8:00PM
 ET, 9/24 5:00PM PT)

Re: Who just force-pushed to apache?

[Eric Newton]

Thanks Josh... apparently I still don't understand the git docs.


On Mon, Sep 22, 2014 at 6:43 PM, Josh Elser josh.el...@gmail.com wrote:

 Ok, I think everything should be back to normal. In summary, no one
 force-pushed to the repository, but the introduction of new commits on
 an old version of a branch that was deleted on the server appears
 identical to a force-push if your copy of the branch was newer.

 * I restored some of the old branches that Eric's push nuked
 (ACCUMULO-652 and ACCUMULO-CURATOR)
 * Re-deleted old branches corresponding to versions that have been
 released or abandoned ({1.4.5, 1.5.1, 1.5.2, 1.6.0}-SNAPSHOT)
 * cherry-picked the change from ACCUMULO-3157 from the 1.5.2-SNAPSHOT
 branch to the 1.5.3-SNAPSHOT branch.

 Please be careful when you push things, especially when using any
 option which is destructive on the server (e.g. --prune). It's good
 practice to use the -n option to git-push to make sure that you're
 actually doing what you *think* you're about to do.

 - Josh

 On Mon, Sep 22, 2014 at 4:29 PM, Josh Elser josh.el...@gmail.com wrote:
  Ok, this wasn't a force push, it just appears like one because
  1.5.2-SNAPSHOT was removed after 1.5.2 was released and then Eric applied
  new changes on top of an old version of 1.5.2-SNAPSHOT.
 
  I'm working on figuring out what's going on. Please, everyone, hold off
 on
  commits for a moment..
 
 
  On 9/22/14, 4:16 PM, Josh Elser wrote:
 
  I see 1.5.2-SNAPSHOT and 1.6.1-SNAPSHOT just got forced updates.
 
* [new branch]  1.4.5-SNAPSHOT - origin/1.4.5-SNAPSHOT
* [new branch]  1.5.1-SNAPSHOT - origin/1.5.1-SNAPSHOT
+ 019ff0a...9167993 1.5.2-SNAPSHOT - origin/1.5.2-SNAPSHOT  (forced
  update)
* [new branch]  1.6.0-SNAPSHOT - origin/1.6.0-SNAPSHOT
  7ee8628..65be6dc  1.6.1-SNAPSHOT - origin/1.6.1-SNAPSHOT
+ c7269de...2f68e07 1.6.2-SNAPSHOT - origin/1.6.2-SNAPSHOT  (forced
  update)
  16ec519..79eb145  master - origin/master
 
  This is a really big no-no-no. I'll be trying to figure out what
  happened and restore history...

Re: [VOTE] Apache Accumulo 1.5.2 RC1

[Eric Newton]

+1
I pulled the source, verified hashes and signature.  Everything built as
expected.  SunnyDay functional test passes.


On Mon, Sep 15, 2014 at 12:24 PM, Josh Elser josh.el...@gmail.com wrote:

 Devs,

 Please consider the following candidate for Apache Accumulo 1.5.2

 Tag: 1.5.2rc1
 SHA1: 039a2c28bdd474805f34ee33f138b009edda6c4c
 Staging Repository: https://repository.apache.org/content/repositories/
 orgapacheaccumulo-1014/

 Source tarball: http://repository.apache.org/content/repositories/
 orgapacheaccumulo-1014/org/apache/accumulo/accumulo/1.5.
 2/accumulo-1.5.2-src.tar.gz
 Binary tarball: http://repository.apache.org/content/repositories/
 orgapacheaccumulo-1014/org/apache/accumulo/accumulo/1.5.
 2/accumulo-1.5.2-bin.tar.gz
 (Append .sha1, .md5 or .asc to download the signature/hash for a
 given artifact.)

 Signing keys available at: https://www.apache.org/dist/accumulo/KEYS

 Over 1.5.1, we have 109 issues resolved https://git-wip-us.apache.org/
 repos/asf?p=accumulo.git;a=blob;f=CHANGES;h=c2892d6e9b1c6c9b96b2a58fc901a7
 6363ece8b0;hb=039a2c28bdd474805f34ee33f138b009edda6c4c

 Testing: all unit and functional tests are passing and ingested 1B entries
 using CI w/ agitation over rc0.

 Vote will be open until Friday, August 19th 12:00AM UTC (8/18 8:00PM ET,
 8/18 5:00PM PT)

 - Josh

Re: Time to release 1.6.1?

[Eric Newton]

+1 for 1.6.1.

There are people testing a recent 1.6 branch at scale (100s of nodes), with
the intent of pushing it to production.

I would rather have a released version in production.

Thanks for volunteering.  Feel free to contact me if you need a hand with
anything.

-Eric


On Wed, Sep 10, 2014 at 1:49 PM, Josh Elser josh.el...@gmail.com wrote:

 Sure that's fine, Corey. Happy to help coordinate things with you.
 *Hopefully* it's not too painful :)


 On 9/10/14, 10:43 AM, Corey Nolet wrote:

 I had posted this to the mailing list originally after a discussion with
 Christopher at the Accumulo Summit hack-a-thon and because I wanted to get
 into the release process to help out.

 Josh, I still wouldn't mind getting together 1.6.1 if that's okay with
 you.
 If nothing else, it would get someone else following the procedures and
 able to do the release.

 On Wed, Sep 10, 2014 at 1:22 PM, Josh Elser josh.el...@gmail.com wrote:

  That's exactly my plan, Christopher. Keith has been the man working on a
 fix for ACCUMULO-1628 which is what I've been spinning on to get 1.5.2
 out
 the door. I want to spend a little time today looking at his patch to
 understand the fix and run some tests myself. Hopefully John can retest
 the
 patch as well since he had an environment that could reproduce the bug.

 Right after we get 1.5.2, I'm happy to work on 1.6.1 as well.

 - Josh


 On 9/10/14, 10:04 AM, Christopher wrote:

  Because of ACCUMULO-2988 (upgrade path from 1.4.x -- 1.6.y, y = 1),
 I'm
 hoping we can revisit this soon. Maybe get 1.5.2 out the door, followed
 by
 1.6.1 right away.


 --
 Christopher L Tubbs II
 http://gravatar.com/ctubbsii

 On Fri, Jun 20, 2014 at 10:30 AM, Keith Turner ke...@deenlo.com
 wrote:

   On Thu, Jun 19, 2014 at 11:46 AM, Josh Elser josh.el...@gmail.com

 wrote:

   I was thinking the same thing, but I also haven't made any strides


  towards

  getting 1.5.2 closer to happening (as I said I'd try to do).

 I still lack physical resources to do the week-long testing as our
 guidelines currently force us to do. I still think this testing is
 excessive if we're actually releasing bug-fixes, but it does

  differentiate

  us from other communities.


  I want to run some CI test because of the changes I made w/ walog.
 I can
 run the test, but I would like to do that as late as possible.   Just
 let
 me know when you are thinking of cutting a release.

 Also, I would like to get 2827 in for the release.



  I'm really not sure how to approach this which is really why I've been
 stalling on it.


 On 6/19/14, 7:18 AM, Mike Drob wrote:

   I'd like to see 1.5.2 released first, just in case there are issues
 we

 discover during that process that need to be addressed. Also, I think
 it
 would be useful to resolve the discussion surrounding upgrades[1]
 before
 releasing.

 [1]:
 http://mail-archives.apache.org/mod_mbox/accumulo-dev/
 201406.mbox/%3CCAGHyZ6LFuwH%3DqGF9JYpitOY9yYDG-
 sop9g6iq57VFPQRnzmyNQ%40mail.gmail.com%3E


 On Thu, Jun 19, 2014 at 8:09 AM, Corey Nolet cjno...@gmail.com
 wrote:

I'd like to start getting a candidate together if there are no

  objections.

 It looks like we have 65 resolved tickets with a fix version of
 1.6.1.

Re: Review Request 25117: ACCUMULO-1957

[Eric Newton]

/TestProxyInstanceOperations.java 
PRE-CREATION 
  test/src/test/java/org/apache/accumulo/proxy/TestProxyReadWrite.java 
PRE-CREATION 
  test/src/test/java/org/apache/accumulo/proxy/TestProxySecurityOperations.java 
PRE-CREATION 
  test/src/test/java/org/apache/accumulo/proxy/TestProxyTableOperations.java 
PRE-CREATION 
  test/src/test/java/org/apache/accumulo/test/ShellServerIT.java 04fdd1c 
  test/src/test/java/org/apache/accumulo/test/functional/DurabilityIT.java 
PRE-CREATION 
  
test/src/test/java/org/apache/accumulo/test/functional/SessionDurabilityIT.java 
PRE-CREATION 

Diff: https://reviews.apache.org/r/25117/diff/


Testing (updated)
---


Thanks,

Eric Newton

Re: Review Request 25117: ACCUMULO-1957

[Eric Newton]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25117/
---

(Updated Sept. 3, 2014, 9:42 p.m.)


Review request for accumulo, Josh Elser, kturner, and John Vines.


Bugs: ACCUMULO-1957
https://issues.apache.org/jira/browse/ACCUMULO-1957


Repository: accumulo


Description
---

Allow tables and update sessions to be configured with different levels of 
durability (WAL sync'ing).


Diffs
-

  core/src/main/java/org/apache/accumulo/core/client/BatchWriterConfig.java 
e2ec22e 
  
core/src/main/java/org/apache/accumulo/core/client/ConditionalWriterConfig.java 
7671c35 
  core/src/main/java/org/apache/accumulo/core/client/Durability.java 
PRE-CREATION 
  
core/src/main/java/org/apache/accumulo/core/client/impl/ConditionalWriterImpl.java
 f5e6dd2 
  
core/src/main/java/org/apache/accumulo/core/client/impl/TabletServerBatchWriter.java
 f2dd980 
  core/src/main/java/org/apache/accumulo/core/client/impl/Writer.java bf226eb 
  core/src/main/java/org/apache/accumulo/core/conf/Property.java 9837867 
  core/src/main/java/org/apache/accumulo/core/conf/PropertyType.java f39a8bd 
  
core/src/main/java/org/apache/accumulo/core/master/thrift/MasterClientService.java
 9c850a8 
  
core/src/main/java/org/apache/accumulo/core/tabletserver/thrift/TDurability.java
 PRE-CREATION 
  
core/src/main/java/org/apache/accumulo/core/tabletserver/thrift/TabletClientService.java
 2ba7674 
  core/src/main/thrift/tabletserver.thrift 25e0b10 
  
examples/simple/src/main/java/org/apache/accumulo/examples/simple/client/ReadWriteExample.java
 a7b288d 
  server/base/src/main/java/org/apache/accumulo/server/init/Initialize.java 
9b952ba 
  server/tserver/src/main/java/org/apache/accumulo/tserver/Mutations.java 
PRE-CREATION 
  server/tserver/src/main/java/org/apache/accumulo/tserver/TabletMutations.java 
e814f0e 
  server/tserver/src/main/java/org/apache/accumulo/tserver/TabletServer.java 
57e3dee 
  server/tserver/src/main/java/org/apache/accumulo/tserver/log/DfsLogger.java 
c01e54a 
  
server/tserver/src/main/java/org/apache/accumulo/tserver/log/TabletServerLogger.java
 26e6891 
  
server/tserver/src/main/java/org/apache/accumulo/tserver/session/ConditionalSession.java
 26668f6 
  
server/tserver/src/main/java/org/apache/accumulo/tserver/session/UpdateSession.java
 bc04a85 
  
server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/CommitSession.java
 6402797 
  server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/Tablet.java 
37950fc 
  
server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/TabletCommitter.java
 a5d197c 
  test/src/main/java/org/apache/accumulo/test/WrongTabletTest.java aeba2e0 
  
test/src/main/java/org/apache/accumulo/test/performance/thrift/NullTserver.java 
6c34172 
  test/src/test/java/org/apache/accumulo/test/functional/DurabilityIT.java 
PRE-CREATION 
  
test/src/test/java/org/apache/accumulo/test/functional/SessionDurabilityIT.java 
PRE-CREATION 

Diff: https://reviews.apache.org/r/25117/diff/


Testing
---

Added DurabilityIT.


Thanks,

Eric Newton

Re: [VOTE] Apache Accumulo 1.5.2 RC0

[Eric Newton]

I verified the signature  hashes on the src.tar.gz, re-built and ran the
unit tests.

I ran the functional tests, and had several timeout failures:

simple.mapreduce.MapReduceTest
simple.table.TableTest
stress.metadataMaxFiles.MetadataMaxFiles
simple.deleterows.DeleteRowsSplitTest

I'm investigating these.


On Wed, Aug 6, 2014 at 8:58 PM, Josh Elser josh.el...@gmail.com wrote:

 Devs,

 Please consider the following candidate for Apache Accumulo 1.5.2

 Tag: 1.5.2rc0
 SHA1: 1f03ba93f8fc8b24b85873d6f024bd05dd102e11
 Staging Repository: https://repository.apache.org/content/repositories/
 orgapacheaccumulo-1012/

 Source tarball: http://repository.apache.org/content/repositories/
 orgapacheaccumulo-1012/org/apache/accumulo/accumulo/1.5.
 2/accumulo-1.5.2-src.tar.gz
 Binary tarball: http://repository.apache.org/content/repositories/
 orgapacheaccumulo-1012/org/apache/accumulo/accumulo/1.5.
 2/accumulo-1.5.2-bin.tar.gz
 (Append .sha1, .md5 or .asc to download the signature/hash for a
 given artifact.)

 Signing keys available at: https://www.apache.org/dist/accumulo/KEYS

 Over 1.5.1, we have 97 issues resolved. You can find them at
 https://issues.apache.org/jira/secure/ReleaseNote.jspa?
 projectId=12312121version=12326272 or https://git-wip-us.apache.org/
 repos/asf?p=accumulo.git;a=commitdiff;h=139bd83cfbe75c382b8fc1c9791394
 c15ee5f771

 Testing: all unit and functional tests are passing. I ran some brief tests
 with continuous ingest (~3B entries ingested with two tservers, killing
 them periodically) and a few Randomwalk modules. Keith told me in IRC today
 that he's running a Continuous Ingest run on EC2.

 This vote will be open until Tuesday, August 12th 12:00AM UTC.

 - Josh

Re: Waiting for [accumulo] balancer to finish

[Eric Newton]

There isn't really a good way.  The test for the faster balancer verifies
that no tablets are offline.

-Eric



On Mon, Jul 14, 2014 at 3:53 PM, Mike Drob mad...@cloudera.com wrote:

 Do we have a programmatic way of asserting that the balancer has finished?
 I'm creating tables and pre-splitting them before doing any loads and
 currently just sleep for a minute hoping that all of the tablets will be
 distributed enough. Is there a better way?

 Mike

Re: Review Request 23458: ACCUMULO-2992 - no more AccumuloConfiguration.getSiteConfiguration

[Eric Newton]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23458/#review47740
---

Ship it!


Ship It!

- Eric Newton


On July 14, 2014, 8:10 p.m., Bill Havanki wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/23458/
 ---
 
 (Updated July 14, 2014, 8:10 p.m.)
 
 
 Review request for accumulo.
 
 
 Bugs: ACCUMULO-2992
 https://issues.apache.org/jira/browse/ACCUMULO-2992
 
 
 Repository: accumulo
 
 
 Description
 ---
 
 Pretty straightforward removal of the long-deprecated static method 
 AccumuloConfiguration.getSiteConfiguration().
 
 
 Diffs
 -
 
   core/src/main/java/org/apache/accumulo/core/conf/AccumuloConfiguration.java 
 6d20cb0 
   core/src/main/java/org/apache/accumulo/core/file/rfile/PrintInfo.java 
 dc54b49 
   
 core/src/main/java/org/apache/accumulo/core/file/rfile/bcfile/PrintInfo.java 
 f387cc2 
   
 core/src/main/java/org/apache/accumulo/core/util/shell/commands/FateCommand.java
  0196baf 
   test/src/test/java/org/apache/accumulo/test/util/CertUtils.java b561600 
 
 Diff: https://reviews.apache.org/r/23458/diff/
 
 
 Testing
 ---
 
 Compiles, unit tests pass (including CertUtilsTest which was modified in the 
 patch).
 
 
 Thanks,
 
 Bill Havanki
 

Re: MiniAccumuloConfig setProperty

[Eric Newton]

We're intentionally keeping the MAC interface small.




On Thu, Jul 10, 2014 at 4:53 PM, John Vines vi...@apache.org wrote:

 In using MAC on 1.6.1-SNAPSHOT, I noticed that the configimpl has a
 setProperty that is not exposed in the general MiniAccumuloConfig. Is this
 intentional or an oversight?

Re: moving rat to a profile?

[Eric Newton]

-1 for removing the rat plugin for default builds
+1 for a simple way to ignore it

I find Christopher's reasoning sound.  I was guilty of committing half the
non-compliant files before the rat check was on by default.

I wish the rat check was less annoying about build trash.
I wish the rat check produced error messages that didn't require me to
consult a report.
I wish the template mechanism in eclipse worked every time so source files
would always get the copyright header by default.

But, automating the check, and moving it to an earlier point in the dev
cycle is a good thing.

-Eric





On Thu, Jul 10, 2014 at 6:09 PM, Christopher ctubb...@apache.org wrote:

 Since this came up in Review (https://reviews.apache.org/r/23391/), I'm
 revisiting this thread.

 In short, it seems there was some confusion on whether or not we had
 reached consensus on enabling rat by default, with rat.ignoreErrors=true
 set to default, and requiring any entities that should be responsible for
 checking legal status of contributions to turn it in (set
 rat.ignoreErrors=false for Jenkins, committers, etc.)

 I think one problem is that there was not actually consensus. This was a
 proposal made previously by Bill Havanki to do the above, after which we
 agreed to minimally making changes to make git clean -df sufficient. But
 there did not appear to be any consensus reached on disabling error
 checking by default, as proposed. Billie had expressed concerns about
 encouraging committers to not take care in checking the legal status of
 their commits, and not only do I share those concerns, I'd also extend my
 concerns to any contributor. This is simply something that everybody
 contributing to ASF should be aware of. Committers have the extra
 responsibility, but even contributors and potential committers should
 understand the contribution requirements, the development process, and have
 put in some minimal amount of thought about whether they have the ability
 to contribute a particular piece of code legally.

 Additionally, I'm concerned about who we'd be easing things for. There's
 clearly an increased risk of commits/contributions without strict checking
 for licenses (we've seen past commits with these errors... just do `git log
 | grep -i license` to see that). So, we're accepting that risk by making
 the easy route one that serves who, exactly? The ASF makes it quite clear
 that SCM is for developers, not end users, and this is an SCM-specific
 problem (specifically branch switching in git).

 So, I do not think there is consensus. There are certainly outstanding
 concerns for disabling the rat check by default. However, I think there is
 room for further discussion and compromise, and I'd love to consider more
 solutions (or discuss the merits of the proposed solutions). For instance,
 one area of compromise I've conceded (as stated in the review) is that I
 think it'd be a good idea to put the rat check in a profile which can be
 optionally disabled, so that it doesn't take the 3 extra seconds to run,
 which could impact some iterative testing (like running one IT at a time,
 or doing automated `git bisect`).




 --
 Christopher L Tubbs II
 http://gravatar.com/ctubbsii


 On Thu, Jun 19, 2014 at 11:15 AM, Christopher ctubb...@apache.org wrote:

  Agreed. That's a minimum.
 
 
  --
  Christopher L Tubbs II
  http://gravatar.com/ctubbsii
 
 
  On Thu, Jun 19, 2014 at 10:54 AM, Bill Havanki 
 bhava...@clouderagovt.com
  wrote:
 
  I've filed ACCUMULO-2927 to make 'git clean -df' sufficient. No matter
 how
  we decide about the rat plugin, I think not requiring -x is a worthwhile
  goal.
 
 
  On Tue, Jun 17, 2014 at 5:43 PM, Christopher ctubb...@apache.org
 wrote:
 
   I agree that instructing users to use this option to modify the build
  isn't
   acceptable and I wouldn't recommend this as a response to users... I
 was
   only stating this as a fact, to point out that a special profile on by
   default with an option to disable isn't needed, since that's the
 current
   behavior.
  
   I'm more interested in the targeted .gitignore with the recommended
 git
   clean -df option without -x. This helps contributors understand build
   tools, makes them aware of the differences between branches, and
 doesn't
   hide problems introduced by switching branches in an obscure error,
 all
   without blowing away their IDE build files. (though switching branches
   often warrants blowing these IDE files away anyway, since different
  modules
   in different branches will be problematic for most IDEs).
  
  
   --
   Christopher L Tubbs II
   http://gravatar.com/ctubbsii
  
  
   On Tue, Jun 17, 2014 at 4:47 PM, Alex Moundalexis 
  al...@clouderagovt.com
   wrote:
  
This kind of response is hardly conducive to prospective
 contributors.
   
We should consider ourselves lucky whenever a contributor provides a
   patch,
let alone runs a build. Expecting a new contributor be fully aware
 of
  the
Apache licensing 

Re: [GitHub] accumulo pull request: ACCUMULO-2943 Fixing failures where no RNG ...

[Eric Newton]

I'm confused... the diff contains updates to the pom for newer dependencies
(jetty, log4j).  Is the patch against 1.6 or master?

I'm not a regular github user (and my git knowledge consists of a bunch of
questions to [~elserj]).  It looks like the patch contains some changes not
strictly related to the RNG provider.

-Eric


On Tue, Jul 1, 2014 at 8:20 AM, haydenmarchant g...@git.apache.org wrote:

 GitHub user haydenmarchant opened a pull request:

 https://github.com/apache/accumulo/pull/11

 ACCUMULO-2943 Fixing failures where no RNG SUN provider

 Both org.apache.accumulo.core.security.crypto.CrypoTest 
 org.apache.accumulo.core.file.rfile.RFileTest have lots of failures
 due to calls to SecureRandom with Random Number Generator Provider
 hard-coded as Sun. The IBM JVM has it's own built in RNG Provider
 called IBMJCE. 2 issues - hard-coded calls to
 SecureRandom.getInstance(algo,SUN) and also default value in
 Property class is SUN.

 Most failures are due to the CryptoModuleParameters instance being
 populated with default value of Crypto Secure RNG Provider, in
 particular, the following line from
 CryptoModelFactory.fillParamsObjectFromStringMap():


 params.setRandomNumberGeneratorProvider(cryptoOpts.get(Property.CRYPTO_SECURE_RNG_PROVIDER.getKey()));

 Since the default as described in Property class for RNG provider
 is SUN, I have made an override mechanism in which a default
 property can be overidden by passing System property of same name.
 Any property with annotation @SystemOverride has this functionality
 enabled. So, when using a JVM which does not have the SUN RNG
 Provider, a system property (-Dcrypto.secure.rng.provider={provname})
 can be added to the parent pom.xml in the surefire plugin definition
 (same location as the max memory for tests).

 In addition, CryptoTest.testCryptoModuleParamsParsing() has been
 changed to read from a separate config file since it just focuses on
  parsing of params and not the actual instantiation of providers etc...

 You can merge this pull request into a Git repository by running:

 $ git pull https://github.com/haydenmarchant/accumulo ACCUMULO-2943

 Alternatively you can review and apply these changes as the patch at:

 https://github.com/apache/accumulo/pull/11.patch

 To close this pull request, make a commit to your master/trunk branch
 with (at least) the following in the commit message:

 This closes #11

 
 commit cc9ab93aa31f517fca4fe7ccfd7caf7160e07603
 Author: haydenmarchant hayden.march...@gmail.com
 Date:   2014-07-01T12:12:32Z

 ACCUMULO-2943 Fixing failures where no RNG SUN provider

 Both org.apache.accumulo.core.security.crypto.CrypoTest 
 org.apache.accumulo.core.file.rfile.RFileTest have lots of failures
 due to calls to SecureRandom with Random Number Generator Provider
 hard-coded as Sun. The IBM JVM has it's own built in RNG Provider
 called IBMJCE. 2 issues - hard-coded calls to
 SecureRandom.getInstance(algo,SUN) and also default value in
 Property class is SUN.

 Most failures are due to the CryptoModuleParameters instance being
 populated with default value of Crypto Secure RNG Provider, in
 particular, the following line from
 CryptoModelFactory.fillParamsObjectFromStringMap():


 params.setRandomNumberGeneratorProvider(cryptoOpts.get(Property.CRYPTO_SECURE_RNG_PROVIDER.getKey()));

 Since the default as described in Property class for RNG provider
 is SUN, I have made an override mechanism in which a default
 property can be overidden by passing System property of same name.
 Any property with annotation @SystemOverride has this functionality
 enabled. So, when using a JVM which does not have the SUN RNG
 Provider, a system property (-Dcrypto.secure.rng.provider={provname})
 can be added to the parent pom.xml in the surefire plugin definition
 (same location as the max memory for tests).

 In addition, CryptoTest.testCryptoModuleParamsParsing() has been
 changed to read from a separate config file so the on/off variants,
 since it just focuses on parsing of params.

 


 ---
 If your project is set up for it, you can reply to this email and have your
 reply appear on GitHub as well. If your project does not have this feature
 enabled and wishes so, or if the feature is enabled but not working, please
 contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
 with INFRA.
 ---

Re: Review Request 22437: ACCUMULO-2887 - ObservableConfiguration

[Eric Newton]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22437/#review45374
---

Ship it!


Ship It!

- Eric Newton


On June 10, 2014, 8:12 p.m., Bill Havanki wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/22437/
 ---
 
 (Updated June 10, 2014, 8:12 p.m.)
 
 
 Review request for accumulo.
 
 
 Bugs: ACCUMULO-2887
 https://issues.apache.org/jira/browse/ACCUMULO-2887
 
 
 Repository: accumulo
 
 
 Description
 ---
 
 The logic for maintaining and notifying observers in NamespaceConfiguration 
 and TableConfiguration is refactored into a new ObservableConfiguration class.
 
 
 Diffs
 -
 
   
 core/src/main/java/org/apache/accumulo/core/conf/ObservableConfiguration.java 
 PRE-CREATION 
   
 core/src/test/java/org/apache/accumulo/core/conf/ObservableConfigurationTest.java
  PRE-CREATION 
   
 server/base/src/main/java/org/apache/accumulo/server/conf/NamespaceConfWatcher.java
  7f3d73e 
   
 server/base/src/main/java/org/apache/accumulo/server/conf/NamespaceConfiguration.java
  eab198e 
   
 server/base/src/main/java/org/apache/accumulo/server/conf/TableConfWatcher.java
  c407309 
   
 server/base/src/main/java/org/apache/accumulo/server/conf/TableConfiguration.java
  909b450 
 
 Diff: https://reviews.apache.org/r/22437/diff/
 
 
 Testing
 ---
 
 Unit tests pass, including the new ObservableConfigurationTest. I was unable 
 to add observer testing for the refactored classes, but that will arrive late 
 (probably under ACCUMULO-2615).
 
 
 Thanks,
 
 Bill Havanki
 

Re: Documentation not correct for Reading Data.

[Eric Newton]

I've created ACCUMULO-2874
https://issues.apache.org/jira/browse/ACCUMULO-2874.

-Eric



On Mon, Jun 9, 2014 at 9:00 AM, Vicky Kak vicky@gmail.com wrote:

 The following piece also looks crazy

 for(EntryKey,Value entry : scan) {
 String row = entry.getKey().getRow();
 Value value = entry.getValue();
 }



 On Mon, Jun 9, 2014 at 6:21 PM, Vicky Kak vicky@gmail.com wrote:

  I have been looking at the Reading Data section here
  http://accumulo.apache.org/1.6/accumulo_user_manual.html#_reading_data
 
  Looking at the code I don't see the fetchFamily method as mentioned in
  this line
 
  scan.fetchFamily(attributes);
 
  Looks to me that the document is not updated or something had recently
  been changed in the API.
 
  I had a look at the history on 1.6.1-Snapshot branch, I can't make out if
  the fectchFamily was renamed from it there
 
 
 https://github.com/apache/accumulo/commits/master/core/src/main/java/org/apache/accumulo/core/client/impl/ScannerOptions.java
 
  It seems the docs contains wrong method name of fetchFamily, wanted to
  confirm before digging further or raising issue.
 
  Thanks,
  Vicky
 

Re: Review Request 22399: ACCUMULO-2876 - use site config for default VolumeManagerImpl

[Eric Newton]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22399/#review45135
---

Ship it!


Ship It!

- Eric Newton


On June 9, 2014, 8:47 p.m., Bill Havanki wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/22399/
 ---
 
 (Updated June 9, 2014, 8:47 p.m.)
 
 
 Review request for accumulo, Eric Newton and Josh Elser.
 
 
 Bugs: ACCUMULO-2876
 https://issues.apache.org/jira/browse/ACCUMULO-2876
 
 
 Repository: accumulo
 
 
 Description
 ---
 
 This changes VolumeManagerImpl.get() to use the site configuration instead of 
 the ZooKeeper-based system configuration. This seems to align better with 
 what is intended, and it also eliminates the risk of an infinite loop, where 
 the system configuration requires an HdfsZooInstance instance ID, which 
 requires a volume manager.
 
 
 Diffs
 -
 
   
 server/base/src/main/java/org/apache/accumulo/server/fs/VolumeManagerImpl.java
  8fe6579 
 
 Diff: https://reviews.apache.org/r/22399/diff/
 
 
 Testing
 ---
 
 All unit tests pass. The fix also rectified the loop in SystemCredentialsTest 
 after refactoring for ACCUMULO-2615.
 
 
 Thanks,
 
 Bill Havanki
 

Re: Review Request 20525: ACCUMULO-2694 Fix handling of tablet migrations for offline tables.

[Eric Newton]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20525/#review44494
---



server/src/main/java/org/apache/accumulo/server/master/balancer/TabletBalancer.java
https://reviews.apache.org/r/20525/#comment78896

Nit: spelling: deferred, can't



- Eric Newton


On May 17, 2014, 9:25 p.m., Sean Busbey wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/20525/
 ---
 
 (Updated May 17, 2014, 9:25 p.m.)
 
 
 Review request for accumulo, Eric Newton and Mike Drob.
 
 
 Bugs: ACCUMULO-2694
 https://issues.apache.org/jira/browse/ACCUMULO-2694
 
 
 Repository: accumulo
 
 
 Description
 ---
 
 ACCUMULO-2694 Fix handling of tablet migrations for offline tables.
 
 * Adds a funtional test that fails due to not rebalancing
 * Fix master to clear migrations when it learns that a table has gone 
 offline
 * Update master to periodically clean up migrations for offline tables
 * Fix balancers to make sure they log if they can't balance.
 
 
 Diffs
 -
 
   server/pom.xml bd61fe6d870449247cde10ae6ed88c98a31657a2 
   server/src/main/java/org/apache/accumulo/server/master/Master.java 
 a2ad2e65e5766d5760199a2c477679f5616a0710 
   
 server/src/main/java/org/apache/accumulo/server/master/balancer/ChaoticLoadBalancer.java
  e14008a8e4caf72133dd9a08b9aab566bb03d862 
   
 server/src/main/java/org/apache/accumulo/server/master/balancer/DefaultLoadBalancer.java
  1fcab4681c32b20d04d8ab9e72d6a61eb6bdddf0 
   
 server/src/main/java/org/apache/accumulo/server/master/balancer/TabletBalancer.java
  69387d365e7207b2faaa91805ca2fe8b432e0d49 
   test/system/auto/stress/migrations.py 
 d07d7a89ddc930f538f736883d1a3ec43d020d00 
 
 Diff: https://reviews.apache.org/r/20525/diff/
 
 
 Testing
 ---
 
 Ran functional test without other changes - failed. After full patch 
 functional test passes.
 
 
 Thanks,
 
 Sean Busbey
 

Re: stats and collectd

[Eric Newton]

I patched up collectd to pull jmx stats but remember to turn on jmx
metrics. Look at my github for any existing code.

-Eric
On May 19, 2014 1:44 PM, Josh Elser josh.el...@gmail.com wrote:

 Yeah, I believe that's correct. It would be cool if we persisted this to a
 table, but it may be better to just use another metrics gathering/storage
 system here rather than reimplement the wheel since we have nothing here
 presently.

 Eric did write an Accumulo backend for OpenTSDB. I don't think that would
 give enough insight to all the types of stuff we track, but I could be
 mistaken :)

 On 5/19/14, 12:45 PM, Arshak Navruzyan wrote:

 Josh,

 Thanks for the clarification.  In terms of the current implementation,
 guessing the full stats are not stored anywhere.  The monitor code just
 polls each tserver and keep the info it collects in memory?Is that
 right?

 Thanks,

 Arshak


 On Mon, May 19, 2014 at 9:09 AM, Josh Elser josh.el...@gmail.com wrote:

  I haven't seen one myself.

 The rest interfaces for collecting metrics is, sadly, known to be
 lacking.
 This aligns with some stated desires to make the rest interface for
 Accumulo metrics a standardized API (which we assert stability guarantees
 on) and then change the monitor to use those directly (instead of doing
 goofy things in the implementation. We have a ticket somewhere for this
 (I
 think Al Krinker has stated some interest in getting involved here).

 I've been curious about what we could get with Graphite. This would be a
 fun experiment.


 On 5/7/14, 9:09 AM, Arshak Navruzyan wrote:

  I was wondering, is there a plugin for Accumulo for collectd?  I noticed
 the current json / xml interfaces don't provide all the stats that the
 monitor app shows.  Perhaps collectd can be used to collect and store
 the
 full set of stats.

Re: SQL layer over Accumulo?

[Eric Newton]

I have taken a quick look at phoenix.  It's baked into HBase-specific
features pretty hard.

It uses coprocessors to do things like create index entries.  This is a
common enough idiom in the HBase community, but not something we've
supported in Accumulo.  In general, you do not want an accumulo Iterator or
Constraint generating data for other tables.

However, a more sophisticated Percolator type implementation (
https://github.com/keith-turner/Accismus) could support index generation
and query transactions.

We could probably re-use a lot of it, but it's not going to be as simple as
changing the classes that talk to the database back-end.

-Eric


On Tue, Apr 29, 2014 at 9:21 AM, Kepner, Jeremy - 0553 - MITLL 
kep...@ll.mit.edu wrote:

 Hi James,
   Can you explain how the SQL layer to HBase works?
 Regards.  -Jeremy

 On Apr 29, 2014, at 1:32 AM, James Taylor jamestay...@apache.org
  wrote:

  Hello,
  Would there be any interest in developing a SQL-layer on top of Accumulo?
  I'm part of the Apache Phoenix project and we've built a similar system
 on
  top of HBase. I wanted to see if there'd be interest on your end at
 working
  with us to generalizing our client and provide in a server that would do
  Accumulo-specific push down in support of a SQL layer. I suspect there's
  enough similarity between HBase and Accumulo that this would be feasible.
  Thanks,
  James

Re: Review Request 20525: ACCUMULO-2694 Fix handling of tablet migrations for offline tables.

[Eric Newton]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20525/#review41013
---



src/server/src/main/java/org/apache/accumulo/server/master/balancer/DefaultLoadBalancer.java
https://reviews.apache.org/r/20525/#comment74372

noservers is constant, should be NO_SERVERS.



src/server/src/main/java/org/apache/accumulo/server/master/balancer/DefaultLoadBalancer.java
https://reviews.apache.org/r/20525/#comment74370

Duplicate code. Create a public class, add a setter for migrations (or 
provide in the ctor).



src/server/src/main/java/org/apache/accumulo/server/master/balancer/DefaultLoadBalancer.java
https://reviews.apache.org/r/20525/#comment74371

At this point balancing has not been performed, it could throw an error and 
balanceSuccessful doesn't really reflect a successful balance call.  Rename to 
resetBalanceError, or move to post-balance.


- Eric Newton


On April 21, 2014, 9:32 p.m., Sean Busbey wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/20525/
 ---
 
 (Updated April 21, 2014, 9:32 p.m.)
 
 
 Review request for accumulo, Eric Newton and Mike Drob.
 
 
 Bugs: ACCUMULO-2694
 https://issues.apache.org/jira/browse/ACCUMULO-2694
 
 
 Repository: accumulo
 
 
 Description
 ---
 
 ACCUMULO-2694 Fix handling of tablet migrations for offline tables.
 
 * Adds a funtional test that fails due to not rebalancing
 * Fix master to clear migrations when it learns that a table has gone 
 offline
 * Update master to periodically clean up migrations for offline tables
 * Fix balancers to make sure they log if they can't balance.
 
 
 Diffs
 -
 
   src/server/pom.xml dbe4fb4 
   src/server/src/main/java/org/apache/accumulo/server/master/Master.java 
 fb7be51 
   
 src/server/src/main/java/org/apache/accumulo/server/master/balancer/ChaoticLoadBalancer.java
  02a4e89 
   
 src/server/src/main/java/org/apache/accumulo/server/master/balancer/DefaultLoadBalancer.java
  4826097 
   
 src/server/src/main/java/org/apache/accumulo/server/master/balancer/TabletBalancer.java
  ad62360 
   test/system/auto/stress/migrations.py d07d7a8 
 
 Diff: https://reviews.apache.org/r/20525/diff/
 
 
 Testing
 ---
 
 Ran functional test without other changes - failed. After full patch 
 functional test passes.
 
 
 Thanks,
 
 Sean Busbey
 

Re: compatibility check between 1.5.x and 1.6.0?

[Eric Newton]

I believe Keith did a mechanical/manual analysis and brought back a couple
of methods/classes.

-Eric


On Tue, Apr 22, 2014 at 8:46 PM, Sean Busbey bus...@cloudera.com wrote:

 Has anyone done a compatibility check for the public API between 1.5.x and
 1.6.0?

 While looking into ACCUMULO-2722 I noticed some changes that might be
 problematic in client.mock and was wondering if anyone did a larger sweep
 already.

 --
 Sean

Re: [VOTE] Accumulo 1.6.0-RC3

[Eric Newton]

-0

+1 for Christopher's Sisyphean effort on creating releases.


On Tue, Apr 22, 2014 at 7:15 PM, Christopher ctubb...@apache.org wrote:

 I'm also going to -1, due to ACCUMULO-2713. I will prepare an RC4
 tomorrow, to run concurrently with this vote, in the (likely) event
 that this vote fails.

 --
 Christopher L Tubbs II
 http://gravatar.com/ctubbsii


 On Tue, Apr 22, 2014 at 6:01 PM, John Vines vi...@apache.org wrote:
  -1
 
 
  On Tue, Apr 22, 2014 at 11:49 AM, Sean Busbey bus...@cloudera.com
 wrote:
 
  -1 due to ACCUMULO-2713
 
 
  On Mon, Apr 21, 2014 at 5:02 PM, Christopher ctubb...@apache.org
 wrote:
 
   Accumulo Developers,
  
   Please consider the following candidate for Accumulo 1.6.0.
  
   Git Commit: 901c35857ce982f2e4a6f609590a04a7b5a1a815
   Branch: 1.6.0-RC3
  
   Staging repo:
  
 
 https://repository.apache.org/content/repositories/orgapacheaccumulo-1009
   Source:
  
 
 https://repository.apache.org/content/repositories/orgapacheaccumulo-1009/org/apache/accumulo/accumulo/1.6.0/accumulo-1.6.0-src.tar.gz
   Binary:
  
 
 https://repository.apache.org/content/repositories/orgapacheaccumulo-1009/org/apache/accumulo/accumulo/1.6.0/accumulo-1.6.0-bin.tar.gz
   (Append .sha1, .md5 or .asc to download the signature/hash for a
   given artifact.)
  
   All artifacts were built and staged with:
   mvn release:prepare  mvn release:perform
  
   Signing keys available at: https://www.apache.org/dist/accumulo/KEYS
  
   Release notes (in progress):
   http://accumulo.apache.org/release_notes/1.6.0
  
   Changes since RC2 (`git log 73fae63..origin/1.6.0-RC3`):
  
   https://issues.apache.org/jira/browse/ACCUMULO-2529
   https://issues.apache.org/jira/browse/ACCUMULO-2671
   https://issues.apache.org/jira/browse/ACCUMULO-2675
   https://issues.apache.org/jira/browse/ACCUMULO-2680
   https://issues.apache.org/jira/browse/ACCUMULO-2682
   https://issues.apache.org/jira/browse/ACCUMULO-2686
   https://issues.apache.org/jira/browse/ACCUMULO-2690
   https://issues.apache.org/jira/browse/ACCUMULO-2695
   https://issues.apache.org/jira/browse/ACCUMULO-2697
   https://issues.apache.org/jira/browse/ACCUMULO-2700
  
   This vote will remain open for 72 hours (3 days), until Thu, April 24,
   22:00 UTC 2014. (That's 6pm EDT.)
  
   [ ] +1 - I have verified and accept...
   [ ] +0 - I have reservations, but not strong enough to vote against...
   [ ] -1 - Because..., I do not accept...
   ... these artifacts as the 1.6.0 release of Apache Accumulo.
  
   Thanks.
  
   P.S. Hint: download the whole staging repo with
   wget -erobots=off -r -l inf -np -nH
  
 
 https://repository.apache.org/content/repositories/orgapacheaccumulo-1009/
   # note the trailing slash is needed
  
  
   --
   Christopher L Tubbs II
   http://gravatar.com/ctubbsii
  
 
 
 
  --
  Sean