[jira] [Comment Edited] (MESOS-2340) Publish JSON in ZK instead of serialized MasterInfo

2015-06-11 Thread Marco Massenzio (JIRA) 

[ 
https://issues.apache.org/jira/browse/MESOS-2340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14582655#comment-14582655
 ] 

Marco Massenzio edited comment on MESOS-2340 at 6/11/15 10:58 PM:
--

r/34687, upon which this review (r/35043) depended upon, has been discarded, so 
I've discarded this one too.

Back to the drawing board.


was (Author: marco-mesos):
r/34687, upon which this review (r/35043) has been discarded, so I've discarded 
this one too.
back to drawing board.

> Publish JSON in ZK instead of serialized MasterInfo
> ---
>
> Key: MESOS-2340
> URL: https://issues.apache.org/jira/browse/MESOS-2340
> Project: Mesos
>  Issue Type: Improvement
>  Components: leader election
>Reporter: Zameer Manji
>Assignee: Marco Massenzio
>
> Currently to discover the master a client needs the ZK node location and 
> access to the MasterInfo protobuf so it can deserialize the binary blob in 
> the node.
> I think it would be nice to publish JSON (like Twitter's ServerSets) so 
> clients are not tied to protobuf to do service discovery.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (MESOS-2340) Publish JSON in ZK instead of serialized MasterInfo

2015-06-03 Thread Marco Massenzio (JIRA) 

[ 
https://issues.apache.org/jira/browse/MESOS-2340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14571829#comment-14571829
 ] 

Marco Massenzio edited comment on MESOS-2340 at 6/3/15 11:33 PM:
-

I now have some working code that "understands" both JSON and binary PBs.
(also, the code to *write* JSON data is pretty trivial - but this won't go in 
until **after** 0.23, correct?)

Need to test with two masters, running old/new versions, locally.
Stay tuned :) 


was (Author: marco-mesos):
I now have some working code that writes JSON to ZK, but _should_ also 
understand binary PBs.
Need to test with two masters, running old/new versions, locally.

Stay tuned :) 

> Publish JSON in ZK instead of serialized MasterInfo
> ---
>
> Key: MESOS-2340
> URL: https://issues.apache.org/jira/browse/MESOS-2340
> Project: Mesos
>  Issue Type: Improvement
>  Components: leader election
>Reporter: Zameer Manji
>Assignee: Marco Massenzio
>
> Currently to discover the master a client needs the ZK node location and 
> access to the MasterInfo protobuf so it can deserialize the binary blob in 
> the node.
> I think it would be nice to publish JSON (like Twitter's ServerSets) so 
> clients are not tied to protobuf to do service discovery.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (MESOS-2340) Publish JSON in ZK instead of serialized MasterInfo

2015-05-28 Thread Benjamin Hindman (JIRA) 

[ 
https://issues.apache.org/jira/browse/MESOS-2340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14562894#comment-14562894
 ] 

Benjamin Hindman edited comment on MESOS-2340 at 5/28/15 2:05 PM:
--

Just wanted to clarify the proposal here. [~xujyan] added labels in the past 
exactly so we could go through deprecation cycles in a "safe" way. In other 
words, the detector doesn't need to try and parse as JSON unless it's labeled 
as JSON (e.g., 'info-json' or something). The deprecation cycle works exactly 
the way [~vinodkone] and [~jvanremoortere] suggested but without the potential 
false positives of trying to parse as JSON when it might not be JSON.


was (Author: benjaminhindman):
Just wanted to clarify the proposal here. We added labels in the past exactly 
so we could go through deprecation cycles in a "safe" way. In other words, 
there is no need to try and parse as JSON and then as protobuf: if it's not 
labeled indicating it is JSON then we parse as protobuf. [~vinodkone], 
[~xujyan], am I missing something here?

> Publish JSON in ZK instead of serialized MasterInfo
> ---
>
> Key: MESOS-2340
> URL: https://issues.apache.org/jira/browse/MESOS-2340
> Project: Mesos
>  Issue Type: Improvement
>Reporter: Zameer Manji
>Assignee: Marco Massenzio
>
> Currently to discover the master a client needs the ZK node location and 
> access to the MasterInfo protobuf so it can deserialize the binary blob in 
> the node.
> I think it would be nice to publish JSON (like Twitter's ServerSets) so 
> clients are not tied to protobuf to do service discovery.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (MESOS-2340) Publish JSON in ZK instead of serialized MasterInfo

2015-05-23 Thread Marco Massenzio (JIRA) 

[ 
https://issues.apache.org/jira/browse/MESOS-2340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14557533#comment-14557533
 ] 

Marco Massenzio edited comment on MESOS-2340 at 5/23/15 9:27 PM:
-

[~haosd...@gmail.com] sure thing!

Please note I've already implemented a couple of utility methods (in 
protobuf_utils.cpp) to convert JSON to/from {{MasterInfo}}
I was putting the finishing touches on them before submitting for review, as 
preparatory work to this: you could find them useful, I'm planning to finalize 
them over the weekend, please have a look.

Can you please update the [Design 
Doc|https://docs.google.com/document/d/1i2pWJaIjnFYhuR-000NG-AC1rFKKrRh3Wn47Y2G6lRE/edit#heading=h.r15ngqq6atyc]
  and make sure that [~vi...@twitter.com] is happy with this approach?


was (Author: marco-mesos):
[~haosd...@gmail.com] sure thing!

Please note I've already implemented a couple of utility methods (in 
protobuf_utils.cpp) to convert JSON to/from {{MasterInfo}}
I was putting the finishing touches on them before submitting for review, as 
preparatory work to this: you could fine them useful, I'm planning to finalize 
them over the weekend, please have a look.

Can you please update the [Design 
Doc|https://docs.google.com/document/d/1i2pWJaIjnFYhuR-000NG-AC1rFKKrRh3Wn47Y2G6lRE/edit#heading=h.r15ngqq6atyc]
  and make sure that [~vi...@twitter.com] is happy with this approach?

> Publish JSON in ZK instead of serialized MasterInfo
> ---
>
> Key: MESOS-2340
> URL: https://issues.apache.org/jira/browse/MESOS-2340
> Project: Mesos
>  Issue Type: Improvement
>Reporter: Zameer Manji
>Assignee: haosdent
>
> Currently to discover the master a client needs the ZK node location and 
> access to the MasterInfo protobuf so it can deserialize the binary blob in 
> the node.
> I think it would be nice to publish JSON (like Twitter's ServerSets) so 
> clients are not tied to protobuf to do service discovery.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (MESOS-2340) Publish JSON in ZK instead of serialized MasterInfo

2015-05-21 Thread Yan Xu (JIRA) 

[ 
https://issues.apache.org/jira/browse/MESOS-2340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14555194#comment-14555194
 ] 

Yan Xu edited comment on MESOS-2340 at 5/21/15 10:50 PM:
-

[~marco-mesos] Since the scheduler only needs to discover the *already elected* 
leading master. Does the following sound simpler?

Add the logic in {{master.cpp}} and have the *leading master* publish a 
_mirroring_ and _ephemeral_ JSON in a separate {{discovery}} ZooKeeper group 
using {{Group.join()}}, after it has determined that it's has become the new 
leader.

- The sequence ID of the leading master in the {{discovery}} group doesn't even 
need to match the one in {{election}} group. Since both ephemeral, they are 
removed at the same time.
- No change necessary in Group, which doesn't know about ProtoBuf/JSON but only 
sees a {{string}}.
- Master knows the definition of MasterInfo ProtoBuf so it's easy to convert it 
to JSON.
- The scheduler watches the {{discovery}} group and picks the ZNode with the 
lowest sequence number as the leading master. However I think there should 
actually be <= 1 member is that group.
- {{election}} and {{discovery}} are logically separated and the {{discovery}} 
group can even be on a different ZK cluster or a different type of service.


was (Author: xujyan):
[~marco-mesos] Since the scheduler only needs to discover the *already elected* 
leading master. Does the following sound simpler?

Add the logic in {{master.cpp}} and have the *leading master* publish a 
_mirroring_ and _ephemeral_ JSON in a separate {{discovery}} ZooKeeper group 
using {{Group.join()}}, after it has determined that it's has become new leader.

- The sequence ID of the leading master in the {{discovery}} group doesn't even 
need to match the one in {{election}} group. Since both ephemeral, they are 
removed at the same time.
- No change necessary in Group, which doesn't know about ProtoBuf/JSON but only 
sees a {{string}}.
- Master knows the definition of MasterInfo ProtoBuf so it's easy to convert it 
to JSON.
- The scheduler watches the {{discovery}} group and picks the ZNode with the 
lowest sequence number as the leading master. However I think there should 
actually be <= 1 member is that group.
- {{election}} and {{discovery}} are logically separated and they can even be 
on a different ZK cluster or a different service.

> Publish JSON in ZK instead of serialized MasterInfo
> ---
>
> Key: MESOS-2340
> URL: https://issues.apache.org/jira/browse/MESOS-2340
> Project: Mesos
>  Issue Type: Improvement
>Reporter: Zameer Manji
>Assignee: haosdent
>
> Currently to discover the master a client needs the ZK node location and 
> access to the MasterInfo protobuf so it can deserialize the binary blob in 
> the node.
> I think it would be nice to publish JSON (like Twitter's ServerSets) so 
> clients are not tied to protobuf to do service discovery.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (MESOS-2340) Publish JSON in ZK instead of serialized MasterInfo

2015-05-19 Thread Marco Massenzio (JIRA) 

[ 
https://issues.apache.org/jira/browse/MESOS-2340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14550958#comment-14550958
 ] 

Marco Massenzio edited comment on MESOS-2340 at 5/19/15 6:30 PM:
-

so, this is the simplest code I could came up with (this needs refining, 
obviously!) (in {{src/zookeeper/group.cpp}}:

{code}
  // if label is not None, this is the MasterInfo being serialized
  if (label.isSome()) {
// TODO: how do we serialize MasterInfo to JSON? we only have the
// raw serialized data here
string json = "{\"value\": \"foobar\"}";
string loc = result + ".json";
string jsonResult;
zk->create(
  loc,
  json,
  acl,
  ZOO_EPHEMERAL,
  &jsonResult);
LOG(INFO) << "Added JSON data to " << jsonResult;
  }
{code}

If I now start the Master, I can see both nodes in the {{/test/json}} folder.
{noformat}
$ ./bin/mesos-master.sh --zk=zk://127.0.0.1:2181/json/test 
--work_dir=/tmp/mesos --quorum=1
{noformat}
{noformat}
[zk: localhost:2181(CONNECTED) 8] ls /json/test 
[log_replicas, info_10, info_10.json]

[zk: localhost:2181(CONNECTED) 9] get /json/test/info_10.json
{"value": "foobar"}
cZxid = 0xe6
ctime = Tue May 19 11:24:55 PDT 2015
mZxid = 0xe6
mtime = Tue May 19 11:24:55 PDT 2015
pZxid = 0xe6
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x14d496680460057
dataLength = 19
numChildren = 0
{noformat}

and there's no .json in {{log_replicas}}.
I would like to get suggestions as to where to "inject" the JSON, as in the 
Group class, we only get the serialized String, not the {{MasterInfo}} PB.
There are obviously way around this, but I'd like to come up with an extensible 
way.


was (Author: marco-mesos):
so, this is the simplest code I could came up with (this needs refining, 
obviously!) (in {{src/zookeeper/group.cpp}}:

{code}
  // if label is not None, this is the MasterInfo being serialized
  if (label.isSome()) {
// TODO: how do we serialize MasterInfo to JSON? we only have the
// raw serialized data here
string json = "{\"value\": \"foobar\"}";
string loc = result + ".json";
string jsonResult;
zk->create(
  loc,
  json,
  acl,
  ZOO_EPHEMERAL,
  &jsonResult);
LOG(INFO) << "Added JSON data to " << jsonResult;
  }
{info}

If I now start the Master, I can see both nodes in the {{/test/json}} folder.
{noformat}
$ ./bin/mesos-master.sh --zk=zk://127.0.0.1:2181/json/test 
--work_dir=/tmp/mesos --quorum=1
{noformat}
{noformat}
[zk: localhost:2181(CONNECTED) 8] ls /json/test 
[log_replicas, info_10, info_10.json]

[zk: localhost:2181(CONNECTED) 9] get /json/test/info_10.json
{"value": "foobar"}
cZxid = 0xe6
ctime = Tue May 19 11:24:55 PDT 2015
mZxid = 0xe6
mtime = Tue May 19 11:24:55 PDT 2015
pZxid = 0xe6
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x14d496680460057
dataLength = 19
numChildren = 0
{noformat}

and there's no .json in {{log_replicas}}.
I would like to get suggestions as to where to "inject" the JSON, as in the 
Group class, we only get the serialized String, not the {{MasterInfo}} PB.
There are obviously way around this, but I'd like to come up with an extensible 
way.

> Publish JSON in ZK instead of serialized MasterInfo
> ---
>
> Key: MESOS-2340
> URL: https://issues.apache.org/jira/browse/MESOS-2340
> Project: Mesos
>  Issue Type: Improvement
>Reporter: Zameer Manji
>Assignee: haosdent
>
> Currently to discover the master a client needs the ZK node location and 
> access to the MasterInfo protobuf so it can deserialize the binary blob in 
> the node.
> I think it would be nice to publish JSON (like Twitter's ServerSets) so 
> clients are not tied to protobuf to do service discovery.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (MESOS-2340) Publish JSON in ZK instead of serialized MasterInfo

2015-05-18 Thread Nikita Vetoshkin (JIRA) 

[ 
https://issues.apache.org/jira/browse/MESOS-2340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14549923#comment-14549923
 ] 

Nikita Vetoshkin edited comment on MESOS-2340 at 5/19/15 6:46 AM:
--

Won't {{multi}} operation (a.k.a transaction) help with simultaneous multiple 
nodes creation?


was (Author: nekto0n):
Won't {{multi}} operation (a.k.a transaction) help with multiple nodes creation 
simultaneously?

> Publish JSON in ZK instead of serialized MasterInfo
> ---
>
> Key: MESOS-2340
> URL: https://issues.apache.org/jira/browse/MESOS-2340
> Project: Mesos
>  Issue Type: Improvement
>Reporter: Zameer Manji
>Assignee: haosdent
>
> Currently to discover the master a client needs the ZK node location and 
> access to the MasterInfo protobuf so it can deserialize the binary blob in 
> the node.
> I think it would be nice to publish JSON (like Twitter's ServerSets) so 
> clients are not tied to protobuf to do service discovery.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (MESOS-2340) Publish JSON in ZK instead of serialized MasterInfo

2015-05-18 Thread Marco Massenzio (JIRA) 

[ 
https://issues.apache.org/jira/browse/MESOS-2340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14549118#comment-14549118
 ] 

Marco Massenzio edited comment on MESOS-2340 at 5/18/15 8:12 PM:
-

The [design 
doc|https://docs.google.com/document/d/1i2pWJaIjnFYhuR-000NG-AC1rFKKrRh3Wn47Y2G6lRE/edit#]
 has almost been finalized: it outlines the current chosen strategy.


was (Author: marco-mesos):
The [design 
doc](https://docs.google.com/document/d/1i2pWJaIjnFYhuR-000NG-AC1rFKKrRh3Wn47Y2G6lRE/edit#)
 has almost been finalized: it outlines the current chosen strategy.

> Publish JSON in ZK instead of serialized MasterInfo
> ---
>
> Key: MESOS-2340
> URL: https://issues.apache.org/jira/browse/MESOS-2340
> Project: Mesos
>  Issue Type: Improvement
>Reporter: Zameer Manji
>Assignee: haosdent
>
> Currently to discover the master a client needs the ZK node location and 
> access to the MasterInfo protobuf so it can deserialize the binary blob in 
> the node.
> I think it would be nice to publish JSON (like Twitter's ServerSets) so 
> clients are not tied to protobuf to do service discovery.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)