[jira] [Commented] (RATIS-334) Implement server membership for LogService Metadata Service

2018-10-24 Thread Josh Elser (JIRA)


[ 
https://issues.apache.org/jira/browse/RATIS-334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16662965#comment-16662965
 ] 

Josh Elser commented on RATIS-334:
--

Had to tweak the protobuf-compiler pom stuff to account for the move (keep 
generated code out of {{src/main/java}}).

TestLogServiceWithGrpc and TestLogServiceWithNetty were failing because of a 
stubbed out implementation. Fixed those by commenting them out.

TestMetaServer is failing for me on OSX. I ran it on a linux box and I think 
Maven killed it after 10 minutes. Not sure if this is working for you, Sergey..

> Implement server membership for LogService Metadata Service
> ---
>
> Key: RATIS-334
> URL: https://issues.apache.org/jira/browse/RATIS-334
> Project: Ratis
>  Issue Type: Sub-task
>  Components: LogService
>Reporter: Josh Elser
>Assignee: Sergey Soldatov
>Priority: Major
> Attachments: RATIS-334-v1.patch, RATIS-334-v3.patch, 
> RATIS-334-wip.patch
>
>
> Half of the Metadata Service for managing LogStreams.
> RATIS-279 is handling the "DDL" operations, and Rajesh suggested that we spin 
> out management of the servers which are available to back these LogStreams in 
> a second task. This is it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (RATIS-334) Implement server membership for LogService Metadata Service

2018-10-24 Thread Josh Elser (JIRA)


[ 
https://issues.apache.org/jira/browse/RATIS-334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16662909#comment-16662909
 ] 

Josh Elser commented on RATIS-334:
--

Thanks for the quick feedback, Sergey. Let's just plan to leave the comments 
there for now to aid rebase as you say.

[~vrodionov], since you said it was OK earlier, I'll commit this one and let 
you rebase on top of it.

Let's plan to get the other things you're working on captured in Jira, Sergey.

> Implement server membership for LogService Metadata Service
> ---
>
> Key: RATIS-334
> URL: https://issues.apache.org/jira/browse/RATIS-334
> Project: Ratis
>  Issue Type: Sub-task
>  Components: LogService
>Reporter: Josh Elser
>Assignee: Sergey Soldatov
>Priority: Major
> Attachments: RATIS-334-v1.patch, RATIS-334-v3.patch, 
> RATIS-334-wip.patch
>
>
> Half of the Metadata Service for managing LogStreams.
> RATIS-279 is handling the "DDL" operations, and Rajesh suggested that we spin 
> out management of the servers which are available to back these LogStreams in 
> a second task. This is it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (RATIS-334) Implement server membership for LogService Metadata Service

2018-10-24 Thread Vladimir Rodionov (JIRA)


[ 
https://issues.apache.org/jira/browse/RATIS-334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16662888#comment-16662888
 ] 

Vladimir Rodionov commented on RATIS-334:
-

Someone (me or Sergey) should stop and wait until the other patch is committed. 
We have 2 conflicting patches (both will require rebasing) at the time.  

> Implement server membership for LogService Metadata Service
> ---
>
> Key: RATIS-334
> URL: https://issues.apache.org/jira/browse/RATIS-334
> Project: Ratis
>  Issue Type: Sub-task
>  Components: LogService
>Reporter: Josh Elser
>Assignee: Sergey Soldatov
>Priority: Major
> Attachments: RATIS-334-v1.patch, RATIS-334-v3.patch, 
> RATIS-334-wip.patch
>
>
> Half of the Metadata Service for managing LogStreams.
> RATIS-279 is handling the "DDL" operations, and Rajesh suggested that we spin 
> out management of the servers which are available to back these LogStreams in 
> a second task. This is it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (RATIS-334) Implement server membership for LogService Metadata Service

2018-10-24 Thread Sergey Soldatov (JIRA)


[ 
https://issues.apache.org/jira/browse/RATIS-334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16662883#comment-16662883
 ] 

Sergey Soldatov commented on RATIS-334:
---

bq. Just remove these from LogStateMachine instead of commenting out?
Ah. Forgot about this one. I commented to make it easier to resolve conflicts 
during rebase and to track what exactly should be removed from the existing 
code in case if someone has objections. 

Just in case, the list of functionality/improvements I'm working on at the 
moment:
1. Restore quorum in case of node failure. When one of the nodes goes down we 
will be able to restore quorum with a new peer automatically. At the moment 
that would happen on slow node message that leader receive if one of the 
followers was unable to proceed transactions for 60 seconds (possible we need 
to adjust that default timeout for something less).
2. Experimenting with the retry policy for meta queries. The default one is 
very aggressive (resend the query if the previous call takes longer than 3 
seconds). Not sure how to better handle that. Yet.

After that, I will start working on the correctness of all responses and 
error/exception handling during internal operations.

> Implement server membership for LogService Metadata Service
> ---
>
> Key: RATIS-334
> URL: https://issues.apache.org/jira/browse/RATIS-334
> Project: Ratis
>  Issue Type: Sub-task
>  Components: LogService
>Reporter: Josh Elser
>Assignee: Sergey Soldatov
>Priority: Major
> Attachments: RATIS-334-v1.patch, RATIS-334-v3.patch, 
> RATIS-334-wip.patch
>
>
> Half of the Metadata Service for managing LogStreams.
> RATIS-279 is handling the "DDL" operations, and Rajesh suggested that we spin 
> out management of the servers which are available to back these LogStreams in 
> a second task. This is it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (RATIS-334) Implement server membership for LogService Metadata Service

2018-10-24 Thread Sergey Soldatov (JIRA)


[ 
https://issues.apache.org/jira/browse/RATIS-334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16662854#comment-16662854
 ] 

Sergey Soldatov commented on RATIS-334:
---

bq. equals(Object), toString(), and hashCode()?
This class is used only to pass the pair  between client 
and meta. Can add it later, but not sure what for.
bq. should we just have an exception thrown here to prevent the "unhandled 
message" case?
accidentally added that to the git :( Was trying to figure out why my test for 
read/write doesn't work (that was the problem that ratis keeps  support 
messages in the same raft log, so the first was some configuration message with 
empty body)
bq. I know you're planning on work around this, but consolidate calls to 
LogServiceFactory.getInstance() in the constructor of LogServiceClient? Ignore 
if you're already re-working it.
I don't want to instantiate LogService nor use the factory at all. My 
suggestion is to move raft client to the LogStream and instantiate the 
instances using raft group that returned by meta service. But would like to get 
an approval from [~vrodionov]
bq. Re-set the interrupted exception and propagate for InterruptedException. 
Unwrap and re-throw the ExecutionException?
Proper error handling is in my TODO list 
bq. Doing some poking around here to make sure there's nothing obvious in the 
"ecosystem" that might default to using .
That's for tests only. I was going to discuss the default ports/etc. 
bq. I'm also wondering – ideally, we can just set 0 and let the service pick an 
ephemeral port. I guess that would require the service to register itself 
somewhere for us to find it? You have an idea about how that should work?
Exactly like zk works. For masters we have to specify the quorum (list of the 
host:port pairs). Workers would register themselves in meta.
bq. This isn't threadsafe. What about making the global currentGroup some 
AtomicReference and then copy it into a local variable when you need it? 
Admittedly, I don't quite grok what this is doing yet
That was a workaround for a deadlock in ratis itself. Actually was planning to 
fix dead lock rather than keep this code :)
bq. Probably a good idea to let this be configurable too (multiple NICs with 
public/private network setups).
That's in my TODO as well ( all builder params will be configurable). 
bq. Heh , switch to InetAddress.getLocalHost().getHostName() for now. I think 
we'll have to provide the ability later on to specify hostnames for quorum 
members (same reason as above).
heh, that part has a long story :) We have the list of the quorum members, but 
for each node, we need to identify itself in this list to properly create 
RaftPeer with a public address. The idea is to have a common configuration 
without node specific elements. So that was added as a workaround to get 
everything working in real cluster scenario with fewer efforts.  I'm planning 
to enhance that with resolving quorum ips and check which belongs to the 
current node.
bq. Can we push down the log dir into LogServiceWorker? 
Of course,  that will be configurable as well as other params for the workers. 
I already mentioned that earlier in the list of things to do. 
bq.   Does it make sense to have some common configuration across the DML and 
DDL services?
well, at least meta quorum id should be common. We also may do it for working 
dir. For ports we may have some default ports for masters and workers and 
ability to specify those values in the common configuration.


 




> Implement server membership for LogService Metadata Service
> ---
>
> Key: RATIS-334
> URL: https://issues.apache.org/jira/browse/RATIS-334
> Project: Ratis
>  Issue Type: Sub-task
>  Components: LogService
>Reporter: Josh Elser
>Assignee: Sergey Soldatov
>Priority: Major
> Attachments: RATIS-334-v1.patch, RATIS-334-v3.patch, 
> RATIS-334-wip.patch
>
>
> Half of the Metadata Service for managing LogStreams.
> RATIS-279 is handling the "DDL" operations, and Rajesh suggested that we spin 
> out management of the servers which are available to back these LogStreams in 
> a second task. This is it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (RATIS-334) Implement server membership for LogService Metadata Service

2018-10-24 Thread Vladimir Rodionov (JIRA)


[ 
https://issues.apache.org/jira/browse/RATIS-334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16662762#comment-16662762
 ] 

Vladimir Rodionov commented on RATIS-334:
-

np, I will rebase RATIS-369 on top of this.

> Implement server membership for LogService Metadata Service
> ---
>
> Key: RATIS-334
> URL: https://issues.apache.org/jira/browse/RATIS-334
> Project: Ratis
>  Issue Type: Sub-task
>  Components: LogService
>Reporter: Josh Elser
>Assignee: Sergey Soldatov
>Priority: Major
> Attachments: RATIS-334-v1.patch, RATIS-334-v3.patch, 
> RATIS-334-wip.patch
>
>
> Half of the Metadata Service for managing LogStreams.
> RATIS-279 is handling the "DDL" operations, and Rajesh suggested that we spin 
> out management of the servers which are available to back these LogStreams in 
> a second task. This is it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (RATIS-334) Implement server membership for LogService Metadata Service

2018-10-23 Thread Josh Elser (JIRA)


[ 
https://issues.apache.org/jira/browse/RATIS-334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16660867#comment-16660867
 ] 

Josh Elser commented on RATIS-334:
--

Either way to me. I can see arguments in either direction.

> Implement server membership for LogService Metadata Service
> ---
>
> Key: RATIS-334
> URL: https://issues.apache.org/jira/browse/RATIS-334
> Project: Ratis
>  Issue Type: Sub-task
>Reporter: Josh Elser
>Assignee: Sergey Soldatov
>Priority: Major
> Attachments: RATIS-334-v1.patch, RATIS-334-v3.patch, 
> RATIS-334-wip.patch
>
>
> Half of the Metadata Service for managing LogStreams.
> RATIS-279 is handling the "DDL" operations, and Rajesh suggested that we spin 
> out management of the servers which are available to back these LogStreams in 
> a second task. This is it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (RATIS-334) Implement server membership for LogService Metadata Service

2018-10-22 Thread Sergey Soldatov (JIRA)


[ 
https://issues.apache.org/jira/browse/RATIS-334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16660012#comment-16660012
 ] 

Sergey Soldatov commented on RATIS-334:
---

bq. Let's spin this out into its own Jira issue if you think it's important to 
fix 
well, I can't see how to use it. So the question is not about the importance, 
but about usability. But if I'm the only one who sees problems there, ignore 
me.  

> Implement server membership for LogService Metadata Service
> ---
>
> Key: RATIS-334
> URL: https://issues.apache.org/jira/browse/RATIS-334
> Project: Ratis
>  Issue Type: Sub-task
>Reporter: Josh Elser
>Assignee: Sergey Soldatov
>Priority: Major
> Attachments: RATIS-334-v1.patch, RATIS-334-v3.patch, 
> RATIS-334-wip.patch
>
>
> Half of the Metadata Service for managing LogStreams.
> RATIS-279 is handling the "DDL" operations, and Rajesh suggested that we spin 
> out management of the servers which are available to back these LogStreams in 
> a second task. This is it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (RATIS-334) Implement server membership for LogService Metadata Service

2018-10-22 Thread Josh Elser (JIRA)


[ 
https://issues.apache.org/jira/browse/RATIS-334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16659968#comment-16659968
 ] 

Josh Elser commented on RATIS-334:
--

{quote}this class is ambiguous
{quote}
Let's spin this out into its own Jira issue if you think it's important to fix 
:)

> Implement server membership for LogService Metadata Service
> ---
>
> Key: RATIS-334
> URL: https://issues.apache.org/jira/browse/RATIS-334
> Project: Ratis
>  Issue Type: Sub-task
>Reporter: Josh Elser
>Assignee: Sergey Soldatov
>Priority: Major
> Attachments: RATIS-334-v1.patch, RATIS-334-v3.patch, 
> RATIS-334-wip.patch
>
>
> Half of the Metadata Service for managing LogStreams.
> RATIS-279 is handling the "DDL" operations, and Rajesh suggested that we spin 
> out management of the servers which are available to back these LogStreams in 
> a second task. This is it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (RATIS-334) Implement server membership for LogService Metadata Service

2018-10-22 Thread Sergey Soldatov (JIRA)


[ 
https://issues.apache.org/jira/browse/RATIS-334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16659879#comment-16659879
 ] 

Sergey Soldatov commented on RATIS-334:
---

[~vrodionov] well, my point is that this class is ambiguous because:
1.  It uses raft client. And the stream class that provides reader/writer uses 
that raft client, so we can say that this client is the one based on Log quorum.
2. Log quorum is created by the meta service during .createLog() call, so this 
method should not be the part of LogService class.
3. createLog() in LogService is a remote call that returns LogStream. But the 
only thing that we create remotely is the raft group which will be used by raft 
client. But as I already mentioned, LogStream actually takes raft client from 
the LogService.


> Implement server membership for LogService Metadata Service
> ---
>
> Key: RATIS-334
> URL: https://issues.apache.org/jira/browse/RATIS-334
> Project: Ratis
>  Issue Type: Sub-task
>Reporter: Josh Elser
>Assignee: Sergey Soldatov
>Priority: Major
> Attachments: RATIS-334-v1.patch, RATIS-334-v3.patch, 
> RATIS-334-wip.patch
>
>
> Half of the Metadata Service for managing LogStreams.
> RATIS-279 is handling the "DDL" operations, and Rajesh suggested that we spin 
> out management of the servers which are available to back these LogStreams in 
> a second task. This is it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (RATIS-334) Implement server membership for LogService Metadata Service

2018-10-22 Thread Josh Elser (JIRA)


[ 
https://issues.apache.org/jira/browse/RATIS-334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16659600#comment-16659600
 ] 

Josh Elser commented on RATIS-334:
--

{quote}I don't quite understand what LogService class is supposed to do. By the 
name, it should be some kind of service, but it's definitely not. It behaves 
like a client, but it has a factory
{quote}
Yes, that was the intent to have it behave like a client. The class LogService 
is the "entrypoint" to using the "Ratis LogService" (this stuff). Don't have 
strong reasons for naming it one way or the other.

> Implement server membership for LogService Metadata Service
> ---
>
> Key: RATIS-334
> URL: https://issues.apache.org/jira/browse/RATIS-334
> Project: Ratis
>  Issue Type: Sub-task
>Reporter: Josh Elser
>Assignee: Sergey Soldatov
>Priority: Major
> Attachments: RATIS-334-v1.patch, RATIS-334-v3.patch, 
> RATIS-334-wip.patch
>
>
> Half of the Metadata Service for managing LogStreams.
> RATIS-279 is handling the "DDL" operations, and Rajesh suggested that we spin 
> out management of the servers which are available to back these LogStreams in 
> a second task. This is it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (RATIS-334) Implement server membership for LogService Metadata Service

2018-10-22 Thread Vladimir Rodionov (JIRA)


[ 
https://issues.apache.org/jira/browse/RATIS-334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16659342#comment-16659342
 ] 

Vladimir Rodionov commented on RATIS-334:
-

LogService looks like a client, agree. I would leave it, look at this as 
FileSystem -> File relationship, [~sergey.soldatov] 

> Implement server membership for LogService Metadata Service
> ---
>
> Key: RATIS-334
> URL: https://issues.apache.org/jira/browse/RATIS-334
> Project: Ratis
>  Issue Type: Sub-task
>Reporter: Josh Elser
>Assignee: Sergey Soldatov
>Priority: Major
> Attachments: RATIS-334-v1.patch, RATIS-334-v3.patch, 
> RATIS-334-wip.patch
>
>
> Half of the Metadata Service for managing LogStreams.
> RATIS-279 is handling the "DDL" operations, and Rajesh suggested that we spin 
> out management of the servers which are available to back these LogStreams in 
> a second task. This is it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (RATIS-334) Implement server membership for LogService Metadata Service

2018-10-22 Thread Sergey Soldatov (JIRA)


[ 
https://issues.apache.org/jira/browse/RATIS-334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16659297#comment-16659297
 ] 

Sergey Soldatov commented on RATIS-334:
---

[~rajeshbabu] I don't quite understand what LogService class is supposed to do. 
By the name, it should be some kind of service, but it's definitely not. It 
behaves like a client, but it has a factory. It has createLog method, but at 
the same time, it uses a single raft quorum ( so it handles a single Log). From 
my point of view, we don't need this class as well as the factory. All we need 
is to move raft client instance to LogStream. (fyi [~vrodionov])
as for the map of logs and pings - that information is persisted by Ratis 
itself. So during the restart, it would read the raft log from the beginning, 
replaying all transactions. I don't use the snapshots yet (haven't had a chance 
to understand how it works).  
some problems that I'm facing now:
1. retry logic in Ratis. If the request takes longer than 3 seconds (i.e. 
delete log - that would run 3 deleteGroup operations for all peers and it takes 
longer than 3 seconds) than ratis client would automatically resend the query. 
and it's not clear what to do in that case. Right now it tries to delete the 
group again, throwing exceptions. 
2. Ping logic should be improved with tracking timeouts, etc. 
3. Need to add messages as well as the logic for node replacement (i.e. in the 
quorum one of the followers went away). 
4. possible some rework in the messages should be done to unify the responses ( 
with exceptions handling). 

> Implement server membership for LogService Metadata Service
> ---
>
> Key: RATIS-334
> URL: https://issues.apache.org/jira/browse/RATIS-334
> Project: Ratis
>  Issue Type: Sub-task
>Reporter: Josh Elser
>Assignee: Sergey Soldatov
>Priority: Major
> Attachments: RATIS-334-v1.patch, RATIS-334-v3.patch, 
> RATIS-334-wip.patch
>
>
> Half of the Metadata Service for managing LogStreams.
> RATIS-279 is handling the "DDL" operations, and Rajesh suggested that we spin 
> out management of the servers which are available to back these LogStreams in 
> a second task. This is it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (RATIS-334) Implement server membership for LogService Metadata Service

2018-10-22 Thread Rajeshbabu Chintaguntla (JIRA)


[ 
https://issues.apache.org/jira/browse/RATIS-334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16659241#comment-16659241
 ] 

Rajeshbabu Chintaguntla commented on RATIS-334:
---

[~sergey.soldatov] I have gone through the patch. Overall it looks good. What 
do you think of making APIs in LogService like createLog to createLogStream 
etc. as they are same as LogServiceClient APIs? As of now you are writing the 
log registration, ping requests in the meta state machine. I have one dount 
that you still need to write the logic to rebuild the map of logname to peers 
by reading the snapshot right on recovery? or it will be handled by Ratis 
automatically?

> Implement server membership for LogService Metadata Service
> ---
>
> Key: RATIS-334
> URL: https://issues.apache.org/jira/browse/RATIS-334
> Project: Ratis
>  Issue Type: Sub-task
>Reporter: Josh Elser
>Assignee: Sergey Soldatov
>Priority: Major
> Attachments: RATIS-334-v1.patch, RATIS-334-v3.patch, 
> RATIS-334-wip.patch
>
>
> Half of the Metadata Service for managing LogStreams.
> RATIS-279 is handling the "DDL" operations, and Rajesh suggested that we spin 
> out management of the servers which are available to back these LogStreams in 
> a second task. This is it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (RATIS-334) Implement server membership for LogService Metadata Service

2018-10-19 Thread Sergey Soldatov (JIRA)


[ 
https://issues.apache.org/jira/browse/RATIS-334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16657325#comment-16657325
 ] 

Sergey Soldatov commented on RATIS-334:
---

[~vrodionov], [~rajeshbabu] can you take a look. 
TestMetaServer#testReadWritetoLog creates a cluster ( 3 masters, 3 workers), 
creates the log, gets the stream, reader, writer, writes a single message and 
attempts to read it.
Using such kind of tests you may debug/test the full end-to-end scenarios.  I'm 
running it on top of RATIS-338 (in case if the patch does not land well without 
it).

> Implement server membership for LogService Metadata Service
> ---
>
> Key: RATIS-334
> URL: https://issues.apache.org/jira/browse/RATIS-334
> Project: Ratis
>  Issue Type: Sub-task
>Reporter: Josh Elser
>Assignee: Sergey Soldatov
>Priority: Major
> Attachments: RATIS-334-v1.patch, RATIS-334-v3.patch, 
> RATIS-334-wip.patch
>
>
> Half of the Metadata Service for managing LogStreams.
> RATIS-279 is handling the "DDL" operations, and Rajesh suggested that we spin 
> out management of the servers which are available to back these LogStreams in 
> a second task. This is it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (RATIS-334) Implement server membership for LogService Metadata Service

2018-10-17 Thread Sergey Soldatov (JIRA)


[ 
https://issues.apache.org/jira/browse/RATIS-334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16653782#comment-16653782
 ] 

Sergey Soldatov commented on RATIS-334:
---

Ah, forgot to mention that Worker's state machine is actually LogService state 
machine, so with RATIS-274 together and few improvements it would be enough for 
POC projects. Those improvements:
a) make Workers work standalone (add JCommander parameters). Add some basic 
scripts for start/stop service. 
b) make that configurable through parameters (are we planning to support hadoop 
conf or a separate configuration file?)
  


> Implement server membership for LogService Metadata Service
> ---
>
> Key: RATIS-334
> URL: https://issues.apache.org/jira/browse/RATIS-334
> Project: Ratis
>  Issue Type: Sub-task
>Reporter: Josh Elser
>Assignee: Sergey Soldatov
>Priority: Major
> Attachments: RATIS-334-v1.patch, RATIS-334-wip.patch
>
>
> Half of the Metadata Service for managing LogStreams.
> RATIS-279 is handling the "DDL" operations, and Rajesh suggested that we spin 
> out management of the servers which are available to back these LogStreams in 
> a second task. This is it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (RATIS-334) Implement server membership for LogService Metadata Service

2018-10-17 Thread Sergey Soldatov (JIRA)


[ 
https://issues.apache.org/jira/browse/RATIS-334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16653766#comment-16653766
 ] 

Sergey Soldatov commented on RATIS-334:
---

This version includes:
1. proto files for both Log and Meta are moved to the logservice package. 
Generated files are not included to avoid reviewing of 800k+ patch, but will be 
included in the final commit. 
2. Meta support: create/get/list/delete operations. 
3. Components: Master, Worker, LogServiceClient. Worker instantiate a single 
State Machine for all Logs (because of the existing architecture from RATIS-274)
4. Error handling for basic failures such as LogNotFoundException, 
LogAlreadyExistException, NoEnoughWorkersException. 
5. LogService mini cluster. How to operate with it take a look at 
TestMetaServer. 

NB: at the moment create/get operation return LogService instances because of 
RATIS-274. Can be easily adjusted to anything else (LogStream ?) that would 
have ratis client.  Also create/get/list implementations were removed (nulled) 
from LogServiceImpl.   

> Implement server membership for LogService Metadata Service
> ---
>
> Key: RATIS-334
> URL: https://issues.apache.org/jira/browse/RATIS-334
> Project: Ratis
>  Issue Type: Sub-task
>Reporter: Josh Elser
>Assignee: Sergey Soldatov
>Priority: Major
> Attachments: RATIS-334-v1.patch, RATIS-334-wip.patch
>
>
> Half of the Metadata Service for managing LogStreams.
> RATIS-279 is handling the "DDL" operations, and Rajesh suggested that we spin 
> out management of the servers which are available to back these LogStreams in 
> a second task. This is it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (RATIS-334) Implement server membership for LogService Metadata Service

2018-10-09 Thread Sergey Soldatov (JIRA)


[ 
https://issues.apache.org/jira/browse/RATIS-334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16644307#comment-16644307
 ] 

Sergey Soldatov commented on RATIS-334:
---

Just in case, to avoid any kind of confusion. The proto file has some 'patches' 
for messages that already in RATIS-279 and will be replaced by them. 

> Implement server membership for LogService Metadata Service
> ---
>
> Key: RATIS-334
> URL: https://issues.apache.org/jira/browse/RATIS-334
> Project: Ratis
>  Issue Type: Sub-task
>Reporter: Josh Elser
>Assignee: Sergey Soldatov
>Priority: Major
> Attachments: RATIS-334-wip.patch
>
>
> Half of the Metadata Service for managing LogStreams.
> RATIS-279 is handling the "DDL" operations, and Rajesh suggested that we spin 
> out management of the servers which are available to back these LogStreams in 
> a second task. This is it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (RATIS-334) Implement server membership for LogService Metadata Service

2018-10-09 Thread Sergey Soldatov (JIRA)


[ 
https://issues.apache.org/jira/browse/RATIS-334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16644227#comment-16644227
 ] 

Sergey Soldatov commented on RATIS-334:
---

Attached wip patch for better understanding how 'master'-'workers' 
communication would work. May require RATIS-338 applied first. DML file added 
just in case. Was unable to find how to import proto file from another module, 
so had to copy some stuff from ratis-proto :( 
TestMetaServer.java is a test that creates cluster with several master nodes, 
number of workers several quorums with default statemachines. To replace it 
with LogService SM  it should replace BaseStateMachine in LogServiceWorker.java 
at line 78. createLog at the moment doesn't return anything, but would return 
raftGroup (working on the reply at the moment).

> Implement server membership for LogService Metadata Service
> ---
>
> Key: RATIS-334
> URL: https://issues.apache.org/jira/browse/RATIS-334
> Project: Ratis
>  Issue Type: Sub-task
>Reporter: Josh Elser
>Assignee: Sergey Soldatov
>Priority: Major
> Attachments: RATIS-334-wip.patch
>
>
> Half of the Metadata Service for managing LogStreams.
> RATIS-279 is handling the "DDL" operations, and Rajesh suggested that we spin 
> out management of the servers which are available to back these LogStreams in 
> a second task. This is it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)