[ https://issues.apache.org/jira/browse/YARN-2678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14173945#comment-14173945 ]
Steve Loughran commented on YARN-2678: -------------------------------------- Gour: I'll do these, as they seem goood and they need to be done before the registry ships with hadoop. w.r.t the header in the ZK nodes, it's to deal with the problems * znodes with 0 bytes of data still have a stated size of 12 bytes. Requiring 16 bytes of header makes it trivial to decide whether or not it has data (makes enuming child records faster) * allows for future expansion to have different record types. Here's what I propose # drop the header # add a {{type}} field to the json # mandate a service record type declaration {code} "type":"ServiceRecord-1.0.0" {code} # declare that the presence of the byte sequence {{"ServiceRecord-1.0.0"}} implies the entry is a sevice record. JSON is UTF-8 encoded, so this matches the value of the {{type}} field. # if the string is present, the entry MUST be parseable as a service record # declare that the absence of the sequence implies that there is no service record there. {code} Parsing/validating becomes one of {code} if len(data)<len("ServiceRecord-1.0.0"): raise NotFound if not contains(data,"ServiceRecord-1.0.0"): raise NotFound if not parse(data) raise InvalidRecord if not valid(parse(data)) raise InvalidRecord {code} That is, if there isn't the string, it is not a parse error, it is simply "not a record" Validation becomes # assert presence of {{"type":"ServiceRecord-1.0.0"}} # forall endpoints, {{valid(endpoint)}} # endpoints are valid if they follow the structure, al elements in the dictionary of an address are simple strings, etc. that's it: all other fields are optional I'll update the .tla file and the code to match > Recommended improvements to Yarn Registry > ----------------------------------------- > > Key: YARN-2678 > URL: https://issues.apache.org/jira/browse/YARN-2678 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, resourcemanager > Reporter: Gour Saha > Assignee: Steve Loughran > > In the process of binding to Slider AM from Slider agent python code here are > some of the items I stumbled upon and would recommend as improvements. > This is how the Slider's registry looks today - > {noformat} > jsonservicerec{ > "description" : "Slider Application Master", > "external" : [ { > "api" : "org.apache.slider.appmaster", > "addressType" : "host/port", > "protocolType" : "hadoop/protobuf", > "addresses" : [ [ "c6408.ambari.apache.org", "34837" ] ] > }, { > "api" : "org.apache.http.UI", > "addressType" : "uri", > "protocolType" : "webui", > "addresses" : [ [ "http://c6408.ambari.apache.org:43314" ] ] > }, { > "api" : "org.apache.slider.management", > "addressType" : "uri", > "protocolType" : "REST", > "addresses" : [ [ > "http://c6408.ambari.apache.org:43314/ws/v1/slider/mgmt" ] ] > }, { > "api" : "org.apache.slider.publisher", > "addressType" : "uri", > "protocolType" : "REST", > "addresses" : [ [ > "http://c6408.ambari.apache.org:43314/ws/v1/slider/publisher" ] ] > }, { > "api" : "org.apache.slider.registry", > "addressType" : "uri", > "protocolType" : "REST", > "addresses" : [ [ > "http://c6408.ambari.apache.org:43314/ws/v1/slider/registry" ] ] > }, { > "api" : "org.apache.slider.publisher.configurations", > "addressType" : "uri", > "protocolType" : "REST", > "addresses" : [ [ > "http://c6408.ambari.apache.org:43314/ws/v1/slider/publisher/slider" ] ] > } ], > "internal" : [ { > "api" : "org.apache.slider.agents.secure", > "addressType" : "uri", > "protocolType" : "REST", > "addresses" : [ [ > "https://c6408.ambari.apache.org:46958/ws/v1/slider/agents" ] ] > }, { > "api" : "org.apache.slider.agents.oneway", > "addressType" : "uri", > "protocolType" : "REST", > "addresses" : [ [ > "https://c6408.ambari.apache.org:57513/ws/v1/slider/agents" ] ] > } ], > "yarn:persistence" : "application", > "yarn:id" : "application_1412974695267_0015" > } > {noformat} > Recommendations: > 1. I would suggest to either remove the string > {color:red}jsonservicerec{color} or if it is desirable to have a non-null > data at all times then loop the string into the json structure as a top-level > attribute to ensure that the registry data is always a valid json document. > 2. The {color:red}addresses{color} attribute is currently a list of list. I > would recommend to convert it to a list of dictionary objects. In the > dictionary object it would be nice to have the host and port portions of > objects of addressType "uri" as separate key-value pairs to avoid parsing on > the client side. The URI should also be retained as a key say "uri" to avoid > clients trying to generate it by concatenating host, port, resource-path, > etc. Here is a proposed structure - > {noformat} > { > ... > "internal" : [ { > "api" : "org.apache.slider.agents.secure", > "addressType" : "uri", > "protocolType" : "REST", > "addresses" : [ > { "uri" : "https://c6408.ambari.apache.org:46958/ws/v1/slider/agents", > "host" : "c6408.ambari.apache.org", > "port": "46958" > } > ] > } > ], > } > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)