On Fri, Feb 3, 2017 at 1:13 PM, Dorian Ellerbe <[email protected]> wrote:
> Great ideas! > > 1) Regarding SWIM: Looks interesting, but I want to understand/research > more. > > 2) +1 for HTTP. Anything to appeal to a larger audience of users and > committers is good, especially if the dev overhead is relatively low. > > 5) Same idea as #2 above; though this will obviously require a larger > effort. Maybe a functional implementation? > > 6) I've been reading a lot about Docker Swarm. We can spin up N instances > of members and configure them in virtually (no pun) anyway we want. There > are some excellent use cases for it and this is certainly one of them. > > On Mon, Jan 30, 2017 at 2:21 PM Edward Capriolo <[email protected]> > wrote: > > > On Mon, Jan 30, 2017 at 1:03 PM, Sandeep More <[email protected]> > > wrote: > > > > > For #3 the problem is a digest is not easy. The reason is that hosts > > > communicate through each other. If there were three nodes A, B, and C, > > and > > > a message was sent from A -> B and then the message was sent from B -> > C. > > > Node B would be able to change the data and the digest. What I want to > do > > > is be able to ensure that the data is verified by A. This would be > > > something like a PGP email. I want to verify that the message is > > unaltered > > > and that it is from a specific sender. > > > > > > SRM: I see, that would be complicated verifying the sender. I was > > thinking > > > of just verifying the signature on the hashes. Assuming we have a > shared > > > secret amongst all the nodes verifying a signature would not be too > > > difficult. Verifying that the data comes from a specific node might be > > > tricky. > > > > > > "It would be really cool if one could choose a custom data (like a > > > String/Long value), I understand that this could be misused and > > > misinterpreted as datastorage so may be there can be stricter limits on > > the > > > size of the custom payload. This might help Apps to integrate Gossip > > > better." > > > > > > I am unclear about what you are saying here. We can already gossip > > > arbitrary data between nodes. > > > > > > SRM: My bad, I missed this, thanks for pointing it out ! > > > > > > https://github.com/apache/incubator-gossip/blob/master/ > > > src/test/java/org/apache/gossip/DataTest.java > > > > > > Thanks, > > > Edward > > > > > > On Mon, Jan 30, 2017 at 12:45 PM, Edward Capriolo < > [email protected] > > > > > > wrote: > > > > > > > On Mon, Jan 30, 2017 at 9:13 AM, Sandeep More <[email protected] > > > > > > wrote: > > > > > > > > > This is exciting ! > > > > > > > > > > #3, #2 and #1 especially look a great value add. > > > > > > > > > > On #3 I think signing the digest would be easier in short run, > > > encrypting > > > > > will mean involving complex keystore/truststore setup. > > > > > > > > > > May be you already covered this in Recipes or elsewhere but still > > > putting > > > > > it here: > > > > > > > > > > It would be really cool if one could choose a custom data (like a > > > > > String/Long value), I understand that this could be misused and > > > > > misinterpreted as datastorage so may be there can be stricter > limits > > on > > > > the > > > > > size of the custom payload. This might help Apps to integrate > Gossip > > > > > better. > > > > > > > > > > Just a thought ! > > > > > > > > > > Best, > > > > > Sandeep > > > > > > > > > > On Sun, Jan 29, 2017 at 10:25 PM, Edward Capriolo < > > > [email protected] > > > > > > > > > > wrote: > > > > > > > > > > > We currently have almost 10 open tickets for features / > > improvements > > > to > > > > > > gossip, many are slated for the next release and we are on our > way > > to > > > > be > > > > > > ahead of schedule with that. > > > > > > > > > > > > I wanted to pick everyone's brain as to where we should aim for. > > > > > > > > > > > > I think some larger general directions are below: > > > > > > > > > > > > 1) SWIM. https://www.cs.cornell.edu/~ > asdas/research/dsn02-swim.pdf > > > > > > > > > > > > This is a fairly large undertaking in terms of codifying the > > protocol > > > > and > > > > > > keeping the project elegant with two Failure Detectors > > > > > > > > > > > > 2) HTTP as transport > > > > > > > > > > > > Having two transports in the codebase is simple. I think this > will > > go > > > > > well > > > > > > HTTP clients do all the session/persistence. Transplanting > Gossip > > to > > > > > live > > > > > > in a webapp wont be too bad but it might involve re-orging the > > > project > > > > a > > > > > > bit into a multi-module maven project. I see a lot of upside here > > > > > > > > > > > > 3) Signed messages ( > > https://issues.apache.org/jira/browse/GOSSIP-47) > > > > > > > > > > > > While I am not a super expert in keystores and such this strikes > me > > > as > > > > > > interesting idea. Mostly because it allows peers to share info > but > > > also > > > > > be > > > > > > able to sign/verify/encrypt info as it moves between peers. I > have > > > > never > > > > > > seen something quite like this so I think it is novel. > > > > > > > > > > > > 4) Recipes > > > > > > > > > > > > Building things like Ephemeral Nodes or Leader Election seem to > be > > a > > > > good > > > > > > fit. Gossip should not be a database so this is a hard line to > > draw. > > > > This > > > > > > will take some research and expertise to implement correctly. > > > > > > > > > > > > 5) Second implementation > > > > > > > > > > > > Originally I planned to build out a second implementation in c, > > node, > > > > or > > > > > > python. This seems less appealing to me at the moment, but if > > Gossip > > > > Java > > > > > > gets too large/complex we will likely miss out window to do this. > > > > > > > > > > > > 6) something massive to spin up N instances in amazon and do > > testing > > > > > > > > > > > > This is something that needs to happen, maybe in two parts. > > > > > > > > > > > > 7) Other ideas? Let them fly. > > > > > > > > > > > > Thanks, > > > > > > Edward > > > > > > > > > > > > > > > > > > > For #3 the problem is a digest is not easy. The reason is that hosts > > > > communicate through each other. If there were three nodes A, B, and > C, > > > and > > > > a message was sent from A -> B and then the message was sent from B > -> > > C. > > > > Node B would be able to change the data and the digest. What I want > to > > do > > > > is be able to ensure that the data is verified by A. This would be > > > > something like a PGP email. I want to verify that the message is > > > unaltered > > > > and that it is from a specific sender. > > > > > > > > "It would be really cool if one could choose a custom data (like a > > > > String/Long value), I understand that this could be misused and > > > > misinterpreted as datastorage so may be there can be stricter limits > on > > > the > > > > size of the custom payload. This might help Apps to integrate Gossip > > > > better." > > > > > > > > I am unclear about what you are saying here. We can already gossip > > > > arbitrary data between nodes. > > > > > > > > https://github.com/apache/incubator-gossip/blob/master/ > > > > src/test/java/org/apache/gossip/DataTest.java > > > > > > > > Thanks, > > > > Edward > > > > > > > > > > > " Verifying that the data comes from a specific node might be tricky." > > > > For this feature to work the cluster nodes with either need one or more > Pre > > Shared Key or they would all have to agree on a (central) Certificate > > Authority that could grant and verify. I would not expect this to be the > > standard mode for most users, but I think it is fairly novel and would be > > interesting to me. > > > For #6 there are likely a few apache projects products producing docker images. I am open to implementation here.
