GitHub user phrocker opened a pull request:
https://github.com/apache/accumulo/pull/199
DO NOT MERGE: PROOF OF CONCEPT
Hello this is a proof of concept that will hopefully begin the discussion
of changing how we request resources.
First of all there are a lot of changes here. That's primarily because I'm
more focused on getting some ideas out there and getting feedback so that I can
provide a solid design to the community. There are commits from 2015 in here
because I took code from a previous commit that I wanted to include.
The basic concept and what I want from this PR is how do we modify the API
so that we can define resources intelligently AND allow users to request users
within some bounds
I spoke to @keith-turner and @ctubbsii today. We can break this up into
the following ideas.
1) Allow configuration to define resource pools ( memory, cpu, etc ) What
will this look like? How do we constrain this?
2) Define a way of segmenting tables/pools. Right now I have included code
I've used elsewhere to define table specific threadpools, but I think we need
to be more fine grained. But how do we define resource groups? Right now I've
created a stubbed out object called ResourceRequest. That needs to be better
defined, which I will do in the coming days and welcome feedback.
3) How do we define limits on user request? Clearly I've modify the thrift
calls so that we can allow scans to create resource requests, but is this a
good idea? If so, should we create a new permission? How do we avoid abuse with
resource requests?
4) Can we eliminate some of our current resource constraints and merge them
into this concept? I.e. can we get rid of max open files? Can we extend this
concept to be aware of resources consumed by compactions?
5) Could we potentially define more concrete critical sections?
Please ignore the badness in the PR -- I can guarantee that it doesn't
compile. I took a vacation and told @keith-turner I'd put this up here because
I think getting ideas now will serve us better. I'm going to clean this up a
great deal in the next few days so that I can have others help on this. I will
attach tickets to this PR as I have a previous ticket that corresponds to this;
however, I'd welcome input into the list above.
I will update this PR tomorrow with cleanup and begin stubbing out portions
from the list above and other ideas that are presented. Bottom line, what's
your sticking point with how tasks run in Accumulo and is there anything we can
do to improve it?
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/phrocker/accumulo-1 master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/accumulo/pull/199.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #199
----
commit bbd2257a001d654a87b433bb387ad9fce402dbb8
Author: phrocker <[email protected]>
Date: 2015-12-31T15:16:49Z
ACCUMULO-3509: Allow cleanup state to be kept to avoid blocking tserver
session sweep
By enabling state ( true/false) from the cleanup method, the change will
avoid blocking
on a scan session being swept. if the session cleanup blocks because a
ScanSession is
still being read, we may block until the ScanBatch returns for that
ScanSession.
The change uses a simple semaphore ( purely because I like the word ) to
attempt acquisition.
If that fails, we return false from the cleanup and reintroduce that
Session back into
the queue to clean up.
commit 3cd07f03104ef8382afd740de82efdfa2e682f73
Author: Marc <[email protected]>
Date: 2017-01-10T01:20:11Z
THIS IS A POC PR
commit 0fcf266560461c04bcdf7511128d7f6c0a182c26
Author: Marc <[email protected]>
Date: 2017-01-10T01:24:29Z
Merge branch 'master' of https://github.com/phrocker/accumulo-1
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
