[jira] [Commented] (YARN-710) Add to ser/deser methods to RecordFactory
[ https://issues.apache.org/jira/browse/YARN-710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13698232#comment-13698232 ] Siddharth Seth commented on YARN-710: - In the unit test, the setters on the ApplicationId aren't meant to be used (will end up throwing exceptions - this is replaced by newInstance in AppliactionId). Don't think getProto() needs to be changed at all in RecordFactoryPBImpl - instead a new getBuilder method should be sufficient. Somewhere along the flow, it looks like the default proto ends up being created - possibly linked to the getProto changes. > Add to ser/deser methods to RecordFactory > - > > Key: YARN-710 > URL: https://issues.apache.org/jira/browse/YARN-710 > Project: Hadoop YARN > Issue Type: Improvement > Components: api >Affects Versions: 2.0.4-alpha >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: YARN-710.patch, YARN-710.patch, YARN-710-wip.patch > > > I order to do things like AMs failover and checkpointing I need to serialize > app IDs, app attempt IDs, containers and/or IDs, resource requests, etc. > Because we are wrapping/hiding the PB implementation from the APIs, we are > hiding the built in PB ser/deser capabilities. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-710) Add to ser/deser methods to RecordFactory
[ https://issues.apache.org/jira/browse/YARN-710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13671758#comment-13671758 ] Siddharth Seth commented on YARN-710: - bq. On the 3rd one, that would require assuming that I could remove 'PBImpl' from the class, add 'Proto' and then I get the proto implementation. Plus package swithing. While the 'PBImpl' postfix is used as a convention already, the 'Proto' and the package are not used as convention, thus I'd leave it as it is. Is it possible to just use the return type on the getProto method, instead of creating an instance ? (Message message = getProto(newRecordInstance(clazz));) Otherwise, patch looks good to me. > Add to ser/deser methods to RecordFactory > - > > Key: YARN-710 > URL: https://issues.apache.org/jira/browse/YARN-710 > Project: Hadoop YARN > Issue Type: Bug > Components: api >Affects Versions: 2.0.4-alpha >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: YARN-710.patch, YARN-710.patch > > > I order to do things like AMs failover and checkpointing I need to serialize > app IDs, app attempt IDs, containers and/or IDs, resource requests, etc. > Because we are wrapping/hiding the PB implementation from the APIs, we are > hiding the built in PB ser/deser capabilities. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-710) Add to ser/deser methods to RecordFactory
[
https://issues.apache.org/jira/browse/YARN-710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13670637#comment-13670637
]
Hadoop QA commented on YARN-710:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12585455/YARN-710.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1036//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1036//console
This message is automatically generated.
> Add to ser/deser methods to RecordFactory
> -
>
> Key: YARN-710
> URL: https://issues.apache.org/jira/browse/YARN-710
> Project: Hadoop YARN
> Issue Type: Bug
> Components: api
>Affects Versions: 2.0.4-alpha
>Reporter: Alejandro Abdelnur
>Assignee: Alejandro Abdelnur
> Attachments: YARN-710.patch, YARN-710.patch
>
>
> I order to do things like AMs failover and checkpointing I need to serialize
> app IDs, app attempt IDs, containers and/or IDs, resource requests, etc.
> Because we are wrapping/hiding the PB implementation from the APIs, we are
> hiding the built in PB ser/deser capabilities.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-710) Add to ser/deser methods to RecordFactory
[ https://issues.apache.org/jira/browse/YARN-710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13664665#comment-13664665 ] Siddharth Seth commented on YARN-710: - bq. bq. Vinod, Sid, given the discussion, are we OK with the current patch, and what is missing is what will be the public API for this? Yep. The public API is missing. Comments on the patch itself. - The new methods should be marked "@Private" - Should a runtime exception be thrown in case of a reflection error. Limit the scope of the IOException to errors reading from / writing to the stream. - I'm not sure if this is possible, but can the Proto class be figured out in a different manner - directly constructing the classname or from the return type of the getProto method, instead of instantiating *PBImpl and relying on what is returned by getProto. bq. If we don't want to support compatibility of disk serialization for some records, how do you identify the supported ones without a tag interface? I would expect the write call to fail if it is not supported the the given record. bq. I'm OK with having an additional public API class to do the ser/deser of 'supported' records. Still I think such class should delegate to the RecordFactory as the record factory knows about the underlaying implementation and can process the ser/deser leveraging the implementation. I don't think tagging is required for the public API. That can just be determined by what is exposed in the helper library. Like you said, this library can make use of the changes being added in this patch. Related, PB offers other means to create recrods - such as Builder.mergeDelimitedFrom. What the patch does is probably what it should be - i.e. leaves the responsibility of reading/writing multiple records to the user, but would like to here your thoughts on using mergeFrom / mergeDelimitedFrom. Alternately, and I'd like to know what others think, does it make sense for these methods to work with Protos directly. Something like ApplicationIdProto serialize(ApplicationId) ApplicationId deserialize(ApplicationIdProto) > Add to ser/deser methods to RecordFactory > - > > Key: YARN-710 > URL: https://issues.apache.org/jira/browse/YARN-710 > Project: Hadoop YARN > Issue Type: Bug > Components: api >Affects Versions: 2.0.4-alpha >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: YARN-710.patch > > > I order to do things like AMs failover and checkpointing I need to serialize > app IDs, app attempt IDs, containers and/or IDs, resource requests, etc. > Because we are wrapping/hiding the PB implementation from the APIs, we are > hiding the built in PB ser/deser capabilities. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-710) Add to ser/deser methods to RecordFactory
[ https://issues.apache.org/jira/browse/YARN-710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13664537#comment-13664537 ] Alejandro Abdelnur commented on YARN-710: - If we don't want to support compatibility of disk serialization for some records, how do you identify the supported ones without a tag interface? I would expect the write call to fail if it is not supported the the given record. I'm OK with having an additional public API class to do the ser/deser of 'supported' records. Still I think such class should delegate to the RecordFactory as the record factory knows about the underlaying implementation and can process the ser/deser leveraging the implementation. > Add to ser/deser methods to RecordFactory > - > > Key: YARN-710 > URL: https://issues.apache.org/jira/browse/YARN-710 > Project: Hadoop YARN > Issue Type: Bug > Components: api >Affects Versions: 2.0.4-alpha >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: YARN-710.patch > > > I order to do things like AMs failover and checkpointing I need to serialize > app IDs, app attempt IDs, containers and/or IDs, resource requests, etc. > Because we are wrapping/hiding the PB implementation from the APIs, we are > hiding the built in PB ser/deser capabilities. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-710) Add to ser/deser methods to RecordFactory
[ https://issues.apache.org/jira/browse/YARN-710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13664508#comment-13664508 ] Vinod Kumar Vavilapalli commented on YARN-710: -- FWIU, Just having the a public util class which selectively serializes/deserializes records is fine. No need for tagging records. We don't want to support compatibility of disk serialization of some records, for e.g. AllocateResponse. And as Sid says, Records/RecordFactory are all YARN private APIs. So, I'd ideally separate this serialization from YARN internal serialization, by creating a parallel factory class for reading and writing records. Instead if you just want to depend on existing stuff by adding methods to RecordFactory, sure. But like mentioned, it is going to be private and not supportable. > Add to ser/deser methods to RecordFactory > - > > Key: YARN-710 > URL: https://issues.apache.org/jira/browse/YARN-710 > Project: Hadoop YARN > Issue Type: Bug > Components: api >Affects Versions: 2.0.4-alpha >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: YARN-710.patch > > > I order to do things like AMs failover and checkpointing I need to serialize > app IDs, app attempt IDs, containers and/or IDs, resource requests, etc. > Because we are wrapping/hiding the PB implementation from the APIs, we are > hiding the built in PB ser/deser capabilities. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-710) Add to ser/deser methods to RecordFactory
[ https://issues.apache.org/jira/browse/YARN-710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13664484#comment-13664484 ] Alejandro Abdelnur commented on YARN-710: - Vinod, Sid, given the discussion, are we OK with the current patch, and what is missing is what will be the public API for this? > Add to ser/deser methods to RecordFactory > - > > Key: YARN-710 > URL: https://issues.apache.org/jira/browse/YARN-710 > Project: Hadoop YARN > Issue Type: Bug > Components: api >Affects Versions: 2.0.4-alpha >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: YARN-710.patch > > > I order to do things like AMs failover and checkpointing I need to serialize > app IDs, app attempt IDs, containers and/or IDs, resource requests, etc. > Because we are wrapping/hiding the PB implementation from the APIs, we are > hiding the built in PB ser/deser capabilities. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-710) Add to ser/deser methods to RecordFactory
[ https://issues.apache.org/jira/browse/YARN-710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13664459#comment-13664459 ] Siddharth Seth commented on YARN-710: - On a related not RecordFactory.newInstance / Records.newInstance will eventually be private, but are still used in MR since that uses the same RPC framework. We'll need to figure this out eventually, whether LimitedPrivate for MR or changing the way MR records are created. > Add to ser/deser methods to RecordFactory > - > > Key: YARN-710 > URL: https://issues.apache.org/jira/browse/YARN-710 > Project: Hadoop YARN > Issue Type: Bug > Components: api >Affects Versions: 2.0.4-alpha >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: YARN-710.patch > > > I order to do things like AMs failover and checkpointing I need to serialize > app IDs, app attempt IDs, containers and/or IDs, resource requests, etc. > Because we are wrapping/hiding the PB implementation from the APIs, we are > hiding the built in PB ser/deser capabilities. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-710) Add to ser/deser methods to RecordFactory
[ https://issues.apache.org/jira/browse/YARN-710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13664454#comment-13664454 ] Alejandro Abdelnur commented on YARN-710: - If you don't have the tag interface, how you'll decide which public API records are 'serializable' ? > Add to ser/deser methods to RecordFactory > - > > Key: YARN-710 > URL: https://issues.apache.org/jira/browse/YARN-710 > Project: Hadoop YARN > Issue Type: Bug > Components: api >Affects Versions: 2.0.4-alpha >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: YARN-710.patch > > > I order to do things like AMs failover and checkpointing I need to serialize > app IDs, app attempt IDs, containers and/or IDs, resource requests, etc. > Because we are wrapping/hiding the PB implementation from the APIs, we are > hiding the built in PB ser/deser capabilities. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-710) Add to ser/deser methods to RecordFactory
[ https://issues.apache.org/jira/browse/YARN-710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13664446#comment-13664446 ] Siddharth Seth commented on YARN-710: - bq. Add public util class to ser/deser tagged record classes that delegates the RecordFactory Is this not enough ? > Add to ser/deser methods to RecordFactory > - > > Key: YARN-710 > URL: https://issues.apache.org/jira/browse/YARN-710 > Project: Hadoop YARN > Issue Type: Bug > Components: api >Affects Versions: 2.0.4-alpha >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: YARN-710.patch > > > I order to do things like AMs failover and checkpointing I need to serialize > app IDs, app attempt IDs, containers and/or IDs, resource requests, etc. > Because we are wrapping/hiding the PB implementation from the APIs, we are > hiding the built in PB ser/deser capabilities. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-710) Add to ser/deser methods to RecordFactory
[ https://issues.apache.org/jira/browse/YARN-710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13664411#comment-13664411 ] Alejandro Abdelnur commented on YARN-710: - So, it seems what we should do is: * Define a public tag interface that indicates that a record can be serialized * Add that the tag interface to all records we consider may need to be serialized by apps * Add public util class to ser/deser tagged record classes that delegates the RecordFactory Does this sound right? > Add to ser/deser methods to RecordFactory > - > > Key: YARN-710 > URL: https://issues.apache.org/jira/browse/YARN-710 > Project: Hadoop YARN > Issue Type: Bug > Components: api >Affects Versions: 2.0.4-alpha >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: YARN-710.patch > > > I order to do things like AMs failover and checkpointing I need to serialize > app IDs, app attempt IDs, containers and/or IDs, resource requests, etc. > Because we are wrapping/hiding the PB implementation from the APIs, we are > hiding the built in PB ser/deser capabilities. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-710) Add to ser/deser methods to RecordFactory
[ https://issues.apache.org/jira/browse/YARN-710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13664384#comment-13664384 ] Siddharth Seth commented on YARN-710: - bq. Regardless of that, the ser/deser mechanism provided in this patch should do the real work for all records. Then it is matter of deciding for which ones we want to expose the ser/deser capabilities proxying to record factory implementation, right? Agreed. This can be the internal (private) implementation for whatever the public utility is that exposes ser/deser for various records. The internal implementation could then be used by RMStateStore as well. > Add to ser/deser methods to RecordFactory > - > > Key: YARN-710 > URL: https://issues.apache.org/jira/browse/YARN-710 > Project: Hadoop YARN > Issue Type: Bug > Components: api >Affects Versions: 2.0.4-alpha >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: YARN-710.patch > > > I order to do things like AMs failover and checkpointing I need to serialize > app IDs, app attempt IDs, containers and/or IDs, resource requests, etc. > Because we are wrapping/hiding the PB implementation from the APIs, we are > hiding the built in PB ser/deser capabilities. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-710) Add to ser/deser methods to RecordFactory
[ https://issues.apache.org/jira/browse/YARN-710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13663858#comment-13663858 ] Alejandro Abdelnur commented on YARN-710: - [~vinodkv], the description of the JIRA states what is the use case. From Yarn public API it is not possible to get hold of the PB instances unless you use private classes and naming assumptions. This is what the attached patch does, with the intention of keeping it private to YARN while exposing a simple write()/read() API. If you have any, I'd like to hear of other alternatives to address this JIRA. > Add to ser/deser methods to RecordFactory > - > > Key: YARN-710 > URL: https://issues.apache.org/jira/browse/YARN-710 > Project: Hadoop YARN > Issue Type: Bug > Components: api >Affects Versions: 2.0.4-alpha >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: YARN-710.patch > > > I order to do things like AMs failover and checkpointing I need to serialize > app IDs, app attempt IDs, containers and/or IDs, resource requests, etc. > Because we are wrapping/hiding the PB implementation from the APIs, we are > hiding the built in PB ser/deser capabilities. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-710) Add to ser/deser methods to RecordFactory
[ https://issues.apache.org/jira/browse/YARN-710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13663846#comment-13663846 ] Vinod Kumar Vavilapalli commented on YARN-710: -- If you are looking for PB only serialization, why can't you directly use the generated proto records? ResourceManager for example uses them to persist data to RMStateStore. Don't see a need for adding this specifically to the records. If you are looking for generic serialization, I'd +1 Sid's proposal to do it as a library which converts to and fro the serialization format. We shouldn't be inventing serdes for this, no use-case per-se internal to YARN itself. > Add to ser/deser methods to RecordFactory > - > > Key: YARN-710 > URL: https://issues.apache.org/jira/browse/YARN-710 > Project: Hadoop YARN > Issue Type: Bug > Components: api >Affects Versions: 2.0.4-alpha >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: YARN-710.patch > > > I order to do things like AMs failover and checkpointing I need to serialize > app IDs, app attempt IDs, containers and/or IDs, resource requests, etc. > Because we are wrapping/hiding the PB implementation from the APIs, we are > hiding the built in PB ser/deser capabilities. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-710) Add to ser/deser methods to RecordFactory
[ https://issues.apache.org/jira/browse/YARN-710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13663756#comment-13663756 ] Alejandro Abdelnur commented on YARN-710: - Regarding compat, all this is directly based on PROTOBUFs which preserve compatibility if properly used. Regardless of that, the ser/deser mechanism provided in this patch should do the real work for all records. Then it is matter of deciding for which ones we want to expose the ser/deser capabilities proxying to record factory implementation, right? > Add to ser/deser methods to RecordFactory > - > > Key: YARN-710 > URL: https://issues.apache.org/jira/browse/YARN-710 > Project: Hadoop YARN > Issue Type: Bug > Components: api >Affects Versions: 2.0.4-alpha >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: YARN-710.patch > > > I order to do things like AMs failover and checkpointing I need to serialize > app IDs, app attempt IDs, containers and/or IDs, resource requests, etc. > Because we are wrapping/hiding the PB implementation from the APIs, we are > hiding the built in PB ser/deser capabilities. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-710) Add to ser/deser methods to RecordFactory
[ https://issues.apache.org/jira/browse/YARN-710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13663649#comment-13663649 ] Siddharth Seth commented on YARN-710: - Wider base to support in terms of compatibility. This is effectively a utility library which is sitting in the YARN code base to help application developers. I think it's better to list out records which would be useful versus records which should not be serialized. > Add to ser/deser methods to RecordFactory > - > > Key: YARN-710 > URL: https://issues.apache.org/jira/browse/YARN-710 > Project: Hadoop YARN > Issue Type: Bug > Components: api >Affects Versions: 2.0.4-alpha >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: YARN-710.patch > > > I order to do things like AMs failover and checkpointing I need to serialize > app IDs, app attempt IDs, containers and/or IDs, resource requests, etc. > Because we are wrapping/hiding the PB implementation from the APIs, we are > hiding the built in PB ser/deser capabilities. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-710) Add to ser/deser methods to RecordFactory
[ https://issues.apache.org/jira/browse/YARN-710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13663557#comment-13663557 ] Alejandro Abdelnur commented on YARN-710: - What is the harm with blanket serialization? > Add to ser/deser methods to RecordFactory > - > > Key: YARN-710 > URL: https://issues.apache.org/jira/browse/YARN-710 > Project: Hadoop YARN > Issue Type: Bug > Components: api >Affects Versions: 2.0.4-alpha >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: YARN-710.patch > > > I order to do things like AMs failover and checkpointing I need to serialize > app IDs, app attempt IDs, containers and/or IDs, resource requests, etc. > Because we are wrapping/hiding the PB implementation from the APIs, we are > hiding the built in PB ser/deser capabilities. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-710) Add to ser/deser methods to RecordFactory
[ https://issues.apache.org/jira/browse/YARN-710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13663536#comment-13663536 ] Siddharth Seth commented on YARN-710: - bq. Sid, what would be 'relevant' Yarn records? I would say, this should be avail for all records. I'm OK moving the write()/read() methods to another class, RecordIOUtils Should a YARN utility provide serialization for records like StartContainerRequest, AllocateResponse (temporary data) ? bq. Mmhhh, or ... as this is highly coupled to the RecordFactory in use. Why not have this in the RecordFactory as the current patch, and as part of YARN-711 a public proxy class is put in place using the current RecordFactory? Then if you use a different RecordFactory, things will work as expected. YARN-711, IMO, is not the best place to do this - that deals with how YARN records are created from their individual arguments. The current implementation is tightly coupled with the RecordFactory being used, but doesn't necessarily need to be. I think we need to decide whether this will be a blanket serialization mechanism for all YARN records, or a selective set. > Add to ser/deser methods to RecordFactory > - > > Key: YARN-710 > URL: https://issues.apache.org/jira/browse/YARN-710 > Project: Hadoop YARN > Issue Type: Bug > Components: api >Affects Versions: 2.0.4-alpha >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: YARN-710.patch > > > I order to do things like AMs failover and checkpointing I need to serialize > app IDs, app attempt IDs, containers and/or IDs, resource requests, etc. > Because we are wrapping/hiding the PB implementation from the APIs, we are > hiding the built in PB ser/deser capabilities. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-710) Add to ser/deser methods to RecordFactory
[ https://issues.apache.org/jira/browse/YARN-710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13663493#comment-13663493 ] Alejandro Abdelnur commented on YARN-710: - Sid, what would be 'relevant' Yarn records? I would say, this should be avail for all records. I'm OK moving the write()/read() methods to another class, RecordIOUtils?. Mmhhh, or ... as this is highly coupled to the RecordFactory in use. Why not have this in the RecordFactory as the current patch, and as part of YARN-711 a public proxy class is put in place using the current RecordFactory? Then if you use a different RecordFactory, things will work as expected. > Add to ser/deser methods to RecordFactory > - > > Key: YARN-710 > URL: https://issues.apache.org/jira/browse/YARN-710 > Project: Hadoop YARN > Issue Type: Bug > Components: api >Affects Versions: 2.0.4-alpha >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: YARN-710.patch > > > I order to do things like AMs failover and checkpointing I need to serialize > app IDs, app attempt IDs, containers and/or IDs, resource requests, etc. > Because we are wrapping/hiding the PB implementation from the APIs, we are > hiding the built in PB ser/deser capabilities. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-710) Add to ser/deser methods to RecordFactory
[ https://issues.apache.org/jira/browse/YARN-710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13663479#comment-13663479 ] Siddharth Seth commented on YARN-710: - Post YARN-711, RecordFactory and Record will likely become private interfaces, since object construction is done via the respective Records. Should this, instead, be a utility (similar to BuilderUtils) to serialize relevant YARN records - which could potentially expose PB directly. That delinks RPC serialization and record serialization used for persistence. > Add to ser/deser methods to RecordFactory > - > > Key: YARN-710 > URL: https://issues.apache.org/jira/browse/YARN-710 > Project: Hadoop YARN > Issue Type: Bug > Components: api >Affects Versions: 2.0.4-alpha >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: YARN-710.patch > > > I order to do things like AMs failover and checkpointing I need to serialize > app IDs, app attempt IDs, containers and/or IDs, resource requests, etc. > Because we are wrapping/hiding the PB implementation from the APIs, we are > hiding the built in PB ser/deser capabilities. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-710) Add to ser/deser methods to RecordFactory
[
https://issues.apache.org/jira/browse/YARN-710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13663278#comment-13663278
]
Hadoop QA commented on YARN-710:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12584052/YARN-710.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/976//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/976//console
This message is automatically generated.
> Add to ser/deser methods to RecordFactory
> -
>
> Key: YARN-710
> URL: https://issues.apache.org/jira/browse/YARN-710
> Project: Hadoop YARN
> Issue Type: Bug
> Components: api
>Affects Versions: 2.0.4-alpha
>Reporter: Alejandro Abdelnur
>Assignee: Alejandro Abdelnur
> Attachments: YARN-710.patch
>
>
> I order to do things like AMs failover and checkpointing I need to serialize
> app IDs, app attempt IDs, containers and/or IDs, resource requests, etc.
> Because we are wrapping/hiding the PB implementation from the APIs, we are
> hiding the built in PB ser/deser capabilities.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-710) Add to ser/deser methods to RecordFactory
[
https://issues.apache.org/jira/browse/YARN-710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13663247#comment-13663247
]
Alejandro Abdelnur commented on YARN-710:
-
Per discussion in the yarn-dev@,
http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201305.mbox/browser ,
as per Bobby suggestion, a simple way to do this would be something along the
following lines:
{code}
public interface RecordFactory {
public T newRecordInstance(Class clazz) throws YarnException;
public T newRecordInstance(Class clazz, byte[] data) throws
YarnException;
public byte[] fromRecordInstance(T t) throws YarnException;
}
{code}
> Add to ser/deser methods to RecordFactory
> -
>
> Key: YARN-710
> URL: https://issues.apache.org/jira/browse/YARN-710
> Project: Hadoop YARN
> Issue Type: Bug
> Components: api
>Affects Versions: 2.0.4-alpha
>Reporter: Alejandro Abdelnur
>Assignee: Alejandro Abdelnur
> Fix For: 2.0.5-beta
>
>
> I order to do things like AMs failover and checkpointing I need to serialize
> app IDs, app attempt IDs, containers and/or IDs, resource requests, etc.
> Because we are wrapping/hiding the PB implementation from the APIs, we are
> hiding the built in PB ser/deser capabilities.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
