keith-turner opened a new pull request, #6119:
URL: https://github.com/apache/accumulo/pull/6119
It has a lot of rough edges, but this branch has a working distributed Fate
where multiple processes can run fate operations. The following are the
highlights.
* ManagerWorker : a new server class that runs fate operations. These are
currently only started by the new IT.
* FateManager : This class partitions fate work across ManagerWorker
processes. Its currently only run by the new IT, would eventually be run by
the manager.
* MultipleManagerIT : probably the only IT that works. It spins up
multiple processes to run fate operations and then starts many threads that do
table operations. The table operations are working and fate operation are
running in many processes.
The foundation here is fairly solid so far. The TODOs are
* integerate w/ existing manager code, its all fairly standalone ATM.
Currently this disables the manager process from running fate operations.
Would need to clean up the existing fate code in the manager.
* get tests working
* decide how these new processes should be run
* Need a RPC broadcast notification mechanism. The manager currently has
an in memory notification system where a fate operation can cause the TGW to
take action and visa versa. Could do something w/ one way RPCs. This could be
an independent set of changes. Not having it just makes things slower.
Need to determine how this will work out. Seems like could do one of the
following.
1. Make ever manager process started attempt to be the primary manager and
also start up manager worker services. So if five manager processes were
started, then one would be primary and all 5 would run fate (possible other
stuff in the future).
2. Keep the manager and secondary manager the same as it currently is.
Have a new manager worker process that user can start.
I am leaning towards `#1` because it seems like a simpler user experience.
Also it handles starting a single manager process better. It also avoid a
secondary manager process that is doing nothing. Although if pursuing `#1`
would like to keep the code clean w/ strong encapsulation between functional
pieces.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
