[jira] [Commented] (HDFS-2660) Turn off TestDFSOverAvroRpc

[Doug Cutting (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-2660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13168866#comment-13168866
 ] 

Doug Cutting commented on HDFS-2660:


There's already a test in common for Avro RPC that tests AvroRpcEngine.  
TestDFSOverAvroRpc tests that HDFS works over AvroRPCEngine.  It did until 
HDFS-2647 was committed.

Moreover the point of the Avro RPC support in Common is to support Avro RPC in 
HDFS.  If that effort is to be abandoned then Avro RPC should be removed from 
Common too.  Why else should we support it?

If you want to just remove all of the Avro RPC stuff from Hadoop it should not 
be hard.  There's AvroRpcEngine, AvroSpecificRpcEngine, TestAvroRpcEngine and 
AvroSpecificTestProtocol in common, TestDfsOverAvroRpc in HDFS, some 
annotations scattered around to help Avro reflection, and some residual stuff 
in YARN.  Mostly you could probably just remove the avro-ipc dependency from 
various poms and then remove all the lines that the compiler complains about.  
That would get 90% of it.  If you don't intend to keep TestDFSOverAvroRpc 
working as you develop HDFS then I think it would be best to remove Avro RPC 
support from Hadoop altogether.

> Turn off TestDFSOverAvroRpc
> ---
>
> Key: HDFS-2660
> URL: https://issues.apache.org/jira/browse/HDFS-2660
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: test
>Reporter: Suresh Srinivas
>Assignee: Suresh Srinivas
> Fix For: 0.24.0
>
> Attachments: HDFS-2660.patch, HDFS-2660.txt
>
>
> With HDFS-2647, protobuf based RPCs are enabled for some of the protocol. 
> With this, Avro RPC based protocol is not working. Avro based RPC needs to be 
> turned on similar to how protobuf based RPCs are done. Until such a time, I 
> propose turning off the test.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2664) Remove TestDFSOverAvroRpc

[Doug Cutting (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-2664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13168811#comment-13168811
 ] 

Doug Cutting commented on HDFS-2664:


I believe that HDFS-2647 is the best place for this discussion.  That patch 
caused this test to fail.  That issue should be reverted and re-committed when 
it can pass tests before it is committed.  That's the normal practice.  If you 
wish to change the contract for RPCEngine, if you wish to remove AvroRpcEngine, 
or however you wish to address it, it should be addressed before you commit 
rather than by committing a patch that causes existing tests to fail.  If 
AvroRPCEngine is no longer of interest, remove it in a separate issue, 
otherwise maintain it so that its tests continue to pass.  Please do not just 
remove the failing tests alone.

> Remove TestDFSOverAvroRpc
> -
>
> Key: HDFS-2664
> URL: https://issues.apache.org/jira/browse/HDFS-2664
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Suresh Srinivas
>Assignee: Suresh Srinivas
> Fix For: 0.24.0
>
>
> With HDFS-2647, HDFS has transitioned to protocol buffers. The server side 
> implementation registers PB.class and a BlockingService as 
> implementation. Client side uses PB.class as the interface. The RPC 
> engine used is protobuf both for the RPC proxy and the server. With this 
> TestDFSOverAvroRpc fails. I propose removing this test.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2660) Turn off TestDFSOverAvroRpc

[Doug Cutting (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-2660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13168809#comment-13168809
 ] 

Doug Cutting commented on HDFS-2660:


> One could get Avro working by just passing suitable implementation for which 
> reflection is useful.

Then please make this change or remove the Avro code.  Please don't just 
disable tests.

> Turn off TestDFSOverAvroRpc
> ---
>
> Key: HDFS-2660
> URL: https://issues.apache.org/jira/browse/HDFS-2660
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: test
>Reporter: Suresh Srinivas
>Assignee: Suresh Srinivas
> Fix For: 0.24.0
>
> Attachments: HDFS-2660.patch, HDFS-2660.txt
>
>
> With HDFS-2647, protobuf based RPCs are enabled for some of the protocol. 
> With this, Avro RPC based protocol is not working. Avro based RPC needs to be 
> turned on similar to how protobuf based RPCs are done. Until such a time, I 
> propose turning off the test.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2660) Turn off TestDFSOverAvroRpc

[Doug Cutting (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-2660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13168603#comment-13168603
 ] 

Doug Cutting commented on HDFS-2660:


> In order to support Avro, the right way to do it is, change the getServer() 
> calls and addProtocol() calls to use WireProtocol and no longer just setting 
> the RPC engine.

What do you mean by WireProtocol here?  The protobuf-generated Java interface?  
That contains, e.g., fields that are not meant to be transmitted but are not 
declared transient.  In other words, it's not a very good specification of a 
wire protocol.

Avro actually includes code that knows how to serialize protobuf structs but 
doesn't yet include protobuf-specific code for protocols 
(http://s.apache.org/MZT).  That could be added so that protobuf becomes the 
'serialization-independent' Hadoop protocol specification instead of using Java 
interfaces as we have before.  I'd welcome that contribution to Avro.  Then 
RPCEngine might then be modified to accept protobuf ServiceDescriptors to 
specify protocols rather than Java interfaces.  Or we could abandon 
serialization-independent RPC as an architectural goal and remove the code 
related to it.

> Turn off TestDFSOverAvroRpc
> ---
>
> Key: HDFS-2660
> URL: https://issues.apache.org/jira/browse/HDFS-2660
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: test
>Reporter: Suresh Srinivas
>Assignee: Suresh Srinivas
> Fix For: 0.24.0
>
> Attachments: HDFS-2660.patch, HDFS-2660.txt
>
>
> With HDFS-2647, protobuf based RPCs are enabled for some of the protocol. 
> With this, Avro RPC based protocol is not working. Avro based RPC needs to be 
> turned on similar to how protobuf based RPCs are done. Until such a time, I 
> propose turning off the test.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2669) Enable protobuf rpc for ClientNamenodeProtocol

[Doug Cutting (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-2669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13168582#comment-13168582
 ] 

Doug Cutting commented on HDFS-2669:


This furthers the breakage of TestDFSOverAvroRpc initiated by HDFS-2647.  
Please resolve that before compounding it.

> Enable protobuf rpc for ClientNamenodeProtocol
> --
>
> Key: HDFS-2669
> URL: https://issues.apache.org/jira/browse/HDFS-2669
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Sanjay Radia
>Assignee: Sanjay Radia
> Attachments: pbEnableClientNN1.patch, pbEnableClientNN4.patch
>
>
> Change the client and servers to use protobuf based RPC 
> ClientNamenodeProtocol.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2647) Enable protobuf RPC for InterDatanodeProtocol, ClientDatanodeProtocol, JournalProtocol and NamenodeProtocol

[Doug Cutting (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-2647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13168581#comment-13168581
 ] 

Doug Cutting commented on HDFS-2647:


This should not have been committed without addressing the fact that it breaks 
TestDfsOverAvroRpc.  Prior to this, NameNodeRpcServer was independent of the 
RPCEngine being used.  One could switch the configured RPCEngine and HDFS would 
still work.  After this HDFS seems to only work with the ProtobufRPCEngine.  
That's not simply 'enabling' protobuf RPC.  That's mandating protobuf RPC.  If 
that's desired then support for other RPC systems should probably be removed, 
e.g., remove TestDfsOverAvroRpc, remove AvroRPCEngine, remove Avro annotations 
on HDFS protocols, perhaps even remove the RPCEngine abstraction, since it 
would no longer perform any useful function.  Either that, or figure out a way 
to make protobuf RPC not mandated, but rather merely the default.

> Enable protobuf RPC for InterDatanodeProtocol, ClientDatanodeProtocol, 
> JournalProtocol and NamenodeProtocol
> ---
>
> Key: HDFS-2647
> URL: https://issues.apache.org/jira/browse/HDFS-2647
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: balancer, data-node, hdfs client, name-node
>Affects Versions: 0.24.0
>Reporter: Suresh Srinivas
>Assignee: Suresh Srinivas
> Attachments: HDFS-2647.txt, HDFS-2647.txt
>
>
> This jira changes the client and servers to use protobuf based RPC for the 
> protocols mentioned in the Summary.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2660) Turn off TestDFSOverAvroRpc

[Doug Cutting (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-2660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13168573#comment-13168573
 ] 

Doug Cutting commented on HDFS-2660:


> Before protobuf conversion, the RPC server had protocol and corresponding 
> implementation was implementation of that interface. With protobuf, the 
> server has been changed to use PB interface.

That sounds like a change that violates the design of the RPCEngine API, which 
was meant to provide a serialization and transport-independent abstraction for 
RPC that the rest of Hadoop builds on.

> Does RPC engine pluggability mean, just change the RPC engine to some other 
> engine and every thing continues work fine?

That was the intent.  If that's no longer desired, then RPCEngine should be 
removed and protobuf should be used directly.  HDFS-2647 and now HDFS-2669 seem 
to be assuming that all RPC in HDFS will use protobuf and only protobuf, since 
they're directly specifying protobuf-specifics in e.g., NameNodeRpcServer.java. 
 That class used to be generic, indepdendent of any particular RPC 
serialization or transport.

Is there no way to constrain protobuf-specifics to ProtobufRpcEngine, as has 
been done for other RPC engines?  If not, why can't we at least condition 
protobuf-specifics to cases where protobuf is actually being used, as in the 
patch that I provided.  In other words, why is that patch insufficient?

> Turn off TestDFSOverAvroRpc
> ---
>
> Key: HDFS-2660
> URL: https://issues.apache.org/jira/browse/HDFS-2660
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: test
>Reporter: Suresh Srinivas
>Assignee: Suresh Srinivas
> Fix For: 0.24.0
>
> Attachments: HDFS-2660.patch, HDFS-2660.txt
>
>
> With HDFS-2647, protobuf based RPCs are enabled for some of the protocol. 
> With this, Avro RPC based protocol is not working. Avro based RPC needs to be 
> turned on similar to how protobuf based RPCs are done. Until such a time, I 
> propose turning off the test.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2660) Turn off TestDFSOverAvroRpc

[Doug Cutting (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-2660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13167962#comment-13167962
 ] 

Doug Cutting commented on HDFS-2660:


Are you suggesting we hardwire Datanode->Namenode RPC to use ProtobufRpcEngine? 
 And would all the clients of these protocols also be so hardwired?  That would 
defeat the purpose of the RPCEngine API.

Currently the design is that RPC.getServer() returns an instance of the 
configured RPCEngine's server, and that RPC.getProxy() returns an instance of 
the configured RPCEngine's client.  TestDFSOverAvroRpc configures the RPCEngine 
to be Avro, starts a MiniDfsCluster, then runs some really simple tests 
(creating a file, then reading it).  Currently that causes all RPCs for those 
tests to go over Avro.  Today this mostly just serves to validate that 
RPCEngine is a pluggable interface.

> Turn off TestDFSOverAvroRpc
> ---
>
> Key: HDFS-2660
> URL: https://issues.apache.org/jira/browse/HDFS-2660
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: test
>Reporter: Suresh Srinivas
>Assignee: Suresh Srinivas
> Fix For: 0.24.0
>
> Attachments: HDFS-2660.patch, HDFS-2660.txt
>
>
> With HDFS-2647, protobuf based RPCs are enabled for some of the protocol. 
> With this, Avro RPC based protocol is not working. Avro based RPC needs to be 
> turned on similar to how protobuf based RPCs are done. Until such a time, I 
> propose turning off the test.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2664) Remove TestDFSOverAvroRpc

[Doug Cutting (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-2664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13167909#comment-13167909
 ] 

Doug Cutting commented on HDFS-2664:


I don't think we should remove the test without removing the code that it tests.

What I meant that you might propose is to remove Avro RPC altogether from 
common and HDFS.  The Avro RPC implementation and this test primarily currently 
serve to keep the RpcEngine AP a pluggable interface, facilitating future 
alternate RPC engines.  If folks don't think RpcEngine pluggability is valuable 
then perhaps we should remove AvroRpcEngine, this test, and even the RpcEngine 
API itself.

> Remove TestDFSOverAvroRpc
> -
>
> Key: HDFS-2664
> URL: https://issues.apache.org/jira/browse/HDFS-2664
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Suresh Srinivas
>Assignee: Suresh Srinivas
> Fix For: 0.24.0
>
>
> With HDFS-2647, HDFS has transitioned to protocol buffers. The server side 
> implementation registers PB.class and a BlockingService as 
> implementation. Client side uses PB.class as the interface. The RPC 
> engine used is protobuf both for the RPC proxy and the server. With this 
> TestDFSOverAvroRpc fails. I propose removing this test.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2660) Turn off TestDFSOverAvroRpc

[Doug Cutting (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-2660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13167882#comment-13167882
 ] 

Doug Cutting commented on HDFS-2660:


In the above, when I wrote 'WritableRpcEngine API' I meant 'RpcEngine API'.  
Sorry for the confusion.

> With protobuf, client uses PB.class as interface with RPCEngine set 
> to protobuf.

When we're not using protobuf it shouldn't add a protobuf-specific interface.  
If it always uses the protobuf-generated interface then hdfs no longer uses a 
serialization independent RPC system through a pluggable API and you might as 
well rip out the RpcEngine abstraction altogether.

If I comment out the two calls to DFSUtil.addPBProtocol() in NameNodeRpcServer 
then TestDFSOverAvroRpc passes for me.

It seems to me that those calls not be made unless the RpcEngine is already set 
to protobuf.  Ideally those calls would not be in the generic code, since 
they're specific to the protobuf RpcEngine.  I think the simplest fix at this 
point would be to change DFSUtil.addPBProtocol to only add the protocols when 
the RPC.Server implementation passed is a protobuf RPCEngine instance.  Also, 
in that method, I don't see the point of setting the engine in the 
configuration since it's too late by then and the configured protocol engine 
should not be changed unilaterally like that.

> Turn off TestDFSOverAvroRpc
> ---
>
> Key: HDFS-2660
> URL: https://issues.apache.org/jira/browse/HDFS-2660
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: test
>Reporter: Suresh Srinivas
>Assignee: Suresh Srinivas
> Fix For: 0.24.0
>
> Attachments: HDFS-2660.txt
>
>
> With HDFS-2647, protobuf based RPCs are enabled for some of the protocol. 
> With this, Avro RPC based protocol is not working. Avro based RPC needs to be 
> turned on similar to how protobuf based RPCs are done. Until such a time, I 
> propose turning off the test.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2660) Turn off TestDFSOverAvroRpc

[Doug Cutting (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-2660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13167809#comment-13167809
 ] 

Doug Cutting commented on HDFS-2660:


> TestDFSOverAvroRpc uses Writable implementation. Given that the 
> implementation underneath now is moved over to Protobuf [ ... ]

I'm confused by that statement.  Has the WritableRpcEngine API been changed 
incompatibly?  Is it no longer possible to re-implement the abstract methods of 
that API to provide a different RPC transport?  AvroRpcEngine both implements 
that API and uses the Writable implementation of it as a tunnel.  It thus 
provides a good test of the API's plugability.

> I propose removing TestDFSOverAvroRpc.

Please propose that as a separate issue.  Thanks!

> Turn off TestDFSOverAvroRpc
> ---
>
> Key: HDFS-2660
> URL: https://issues.apache.org/jira/browse/HDFS-2660
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: test
>Reporter: Suresh Srinivas
>Assignee: Suresh Srinivas
> Fix For: 0.24.0
>
> Attachments: HDFS-2660.txt
>
>
> With HDFS-2647, protobuf based RPCs are enabled for some of the protocol. 
> With this, Avro RPC based protocol is not working. Avro based RPC needs to be 
> turned on similar to how protobuf based RPCs are done. Until such a time, I 
> propose turning off the test.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2647) Enable protobuf RPC for InterDatanodeProtocol, ClientDatanodeProtocol, JournalProtocol and NamenodeProtocol

[Doug Cutting (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-2647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13167790#comment-13167790
 ] 

Doug Cutting commented on HDFS-2647:


> However the TestDfsOverAvroRpc is related to this patch.

I think this is caused by:

+DFSUtil.addPBProtocol(conf, InterDatanodeProtocolPB.class, service,
+ipcServer);

Avro reflection then fails when trying to infer a protocol for the PB-generated 
superclass of this.

> Enable protobuf RPC for InterDatanodeProtocol, ClientDatanodeProtocol, 
> JournalProtocol and NamenodeProtocol
> ---
>
> Key: HDFS-2647
> URL: https://issues.apache.org/jira/browse/HDFS-2647
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: balancer, data-node, hdfs client, name-node
>Affects Versions: 0.24.0
>Reporter: Suresh Srinivas
>Assignee: Suresh Srinivas
> Attachments: HDFS-2647.txt, HDFS-2647.txt
>
>
> This jira changes the client and servers to use protobuf based RPC for the 
> protocols mentioned in the Summary.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2660) Turn off TestDFSOverAvroRpc

[Doug Cutting (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-2660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13167727#comment-13167727
 ] 

Doug Cutting commented on HDFS-2660:


I'm confused.  HDFS-2647 broke TestDFSOverAvroRpc, so now the fix is to disable 
the test?  Things should pass tests before we commit them to trunk.  Rather 
than committing this patch you should revert HDFS-2647 and commit it once it 
passes tests.

We should either maintain the Avro RPC code or remove it if we don't intend to 
maintain it.  This is the third time this has happened recently, where code is 
committed even though it causes TestDFSOverAvroRpc to start failing.

If you don't want to maintain the Avro RPC code and intend to break it, then I 
suggest you file an issue to remove it and then make HDFS-2647 depend on that 
issue.  Or else fix TestDFSOverAvroRpc in HDFS-2647.



> Turn off TestDFSOverAvroRpc
> ---
>
> Key: HDFS-2660
> URL: https://issues.apache.org/jira/browse/HDFS-2660
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: test
>Reporter: Suresh Srinivas
>Assignee: Suresh Srinivas
> Fix For: 0.24.0
>
> Attachments: HDFS-2660.txt
>
>
> With HDFS-2647, protobuf based RPCs are enabled for some of the protocol. 
> With this, Avro RPC based protocol is not working. Avro based RPC needs to be 
> turned on similar to how protobuf based RPCs are done. Until such a time, I 
> propose turning off the test.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2643) Update doc versions from 0.20 to 1.0

[Doug Cutting (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-2643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13165370#comment-13165370
 ] 

Doug Cutting commented on HDFS-2643:


FYI, it is possible to have Forrest insert a version specified elsewhere in the 
build by:
 - writing a file defining an entity;
 - including that entity definition in the DOCTYPE of the Forrest XML file
 - referring to the entity defined in the XML

For example, see:

http://svn.apache.org/viewvc/avro/trunk/doc/build.xml?view=markup
http://svn.apache.org/viewvc/avro/trunk/doc/src/content/xdocs/tabs.xml?view=markup

Then the documentation version never gets out of date.


> Update doc versions from 0.20 to 1.0
> 
>
> Key: HDFS-2643
> URL: https://issues.apache.org/jira/browse/HDFS-2643
> Project: Hadoop HDFS
>  Issue Type: Task
>  Components: documentation
>Affects Versions: 1.0.0, 1.1.0
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Tsz Wo (Nicholas), SZE
> Attachments: h2643_20111207_branch-1.patch
>
>
> The docs version is still 0.20.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2625) HA: TestDfsOverAvroRpc failing after introduction of HeartbeatResponse type

[Doug Cutting (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-2625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13163123#comment-13163123
 ] 

Doug Cutting commented on HDFS-2625:


FYI, an alternative would have been to add @Nullable to the method in the 
interface.

> HA: TestDfsOverAvroRpc failing after introduction of HeartbeatResponse type
> ---
>
> Key: HDFS-2625
> URL: https://issues.apache.org/jira/browse/HDFS-2625
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: ha, test
>Affects Versions: HA branch (HDFS-1623)
>Reporter: Todd Lipcon
>Assignee: Todd Lipcon
>Priority: Minor
> Fix For: HA branch (HDFS-1623)
>
> Attachments: hdfs-2625.txt
>
>
> TestDfsOverAvroRpc is failing after HDFS-2616. The issue seems to be that we 
> sometimes fill in the "commands" list in HeartbeatResponse with null to 
> indicate no commands. This makes avro barf with a cryptic message about 
> unions of nulls of unions of arrays of nulls or something.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2578) Use protoc to generate protobuf artifacts instead of checking it in.

[Doug Cutting (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-2578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13154562#comment-13154562
 ] 

Doug Cutting commented on HDFS-2578:


This requires that everyone who builds and every Jenkins box has the right 
version of protoc installed, a version that matches the protobuf java library 
in use.  I suggest testing this explicitly in the build, by checking 'protoc 
--version' against the value in Hadoop's pom.xml, failing the build when they 
don't match.

> Use protoc to generate protobuf artifacts instead of checking it in.
> 
>
> Key: HDFS-2578
> URL: https://issues.apache.org/jira/browse/HDFS-2578
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Affects Versions: 0.24.0
>Reporter: Suresh Srinivas
>Assignee: Suresh Srinivas
> Fix For: 0.24.0
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2532) TestDfsOverAvroRpc timing out in trunk

[Doug Cutting (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-2532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13142579#comment-13142579
 ] 

Doug Cutting commented on HDFS-2532:


Binary search of commits points to r1179877 (HDFS-2181).  This test works if 
you update to r1179876 but hangs when you update to r1179877.  I can start to 
look at why.

> TestDfsOverAvroRpc timing out in trunk
> --
>
> Key: HDFS-2532
> URL: https://issues.apache.org/jira/browse/HDFS-2532
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: test
>Affects Versions: 0.24.0
>Reporter: Todd Lipcon
>Priority: Critical
> Attachments: hdfs-2532-make-timeout.txt
>
>
> "java.io.IOException: java.io.IOException: Unknown protocol: 
> org.apache.hadoop.ipc.AvroRpcEngine$TunnelProtocol" occurs while starting up 
> the DN, and then it hangs waiting for the MiniCluster to start.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2532) TestDfsOverAvroRpc timing out in trunk

[Doug Cutting (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-2532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13142443#comment-13142443
 ] 

Doug Cutting commented on HDFS-2532:


It would be good to figure out what commit broke this.

> TestDfsOverAvroRpc timing out in trunk
> --
>
> Key: HDFS-2532
> URL: https://issues.apache.org/jira/browse/HDFS-2532
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: test
>Affects Versions: 0.24.0
>Reporter: Todd Lipcon
>Priority: Critical
> Attachments: hdfs-2532-make-timeout.txt
>
>
> "java.io.IOException: java.io.IOException: Unknown protocol: 
> org.apache.hadoop.ipc.AvroRpcEngine$TunnelProtocol" occurs while starting up 
> the DN, and then it hangs waiting for the MiniCluster to start.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2511) Add dev script to generate HDFS protobufs

[Doug Cutting (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-2511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13138975#comment-13138975
 ] 

Doug Cutting commented on HDFS-2511:


FWIW, in Avro I added these in Maven.

 http://s.apache.org/5Sh




> Add dev script to generate HDFS protobufs
> -
>
> Key: HDFS-2511
> URL: https://issues.apache.org/jira/browse/HDFS-2511
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: build
>Affects Versions: 0.23.0
>Reporter: Todd Lipcon
>Assignee: Todd Lipcon
>Priority: Minor
> Attachments: hdfs-2511.txt
>
>
> Would like to add a simple shell script to re-generate the protobuf code in 
> HDFS -- just easier than remembering the right syntax.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2298) TestDfsOverAvroRpc is failing on trunk

[Doug Cutting (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-2298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13138768#comment-13138768
 ] 

Doug Cutting commented on HDFS-2298:


As the above reports from Jenkins show, merging this patch alone will not cause 
TestDfsOverAvroRpc to stop failing.  It does make the protocols compatible, 
which is good.  But if we want this test to pass (the actual point of this 
issue) then we need to also merge the other two issues I've named above.

> TestDfsOverAvroRpc is failing on trunk
> --
>
> Key: HDFS-2298
> URL: https://issues.apache.org/jira/browse/HDFS-2298
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: test
>Reporter: Aaron T. Myers
>Assignee: Doug Cutting
> Fix For: 0.23.0, 0.24.0
>
> Attachments: HDFS-2298.patch, HDFS-2298.patch, HDFS-2298.patch, 
> HDFS-2298.patch, HDFS-2298.patch, HDFS-2298.patch, HDFS-2298.patch
>
>
> The relevant bit of the error:
> {noformat}
> ---
> Test set: org.apache.hadoop.hdfs.TestDfsOverAvroRpc
> ---
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.486 sec <<< 
> FAILURE!
> testWorkingDirectory(org.apache.hadoop.hdfs.TestDfsOverAvroRpc)  Time 
> elapsed: 1.424 sec  <<< ERROR!
> org.apache.avro.AvroTypeException: Two methods with same name: delete
> {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2298) TestDfsOverAvroRpc is failing on trunk

[Doug Cutting (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-2298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137481#comment-13137481
 ] 

Doug Cutting commented on HDFS-2298:


Nicholas, so you've elected to merge this without HADOOP-7524 and HADOOP-7693?  
That will synchronize the protocols but won't fix test failures in 0.23.  Is 
that your intent?  It would be much better to present such a proposal before 
committing.

> TestDfsOverAvroRpc is failing on trunk
> --
>
> Key: HDFS-2298
> URL: https://issues.apache.org/jira/browse/HDFS-2298
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: test
>Reporter: Aaron T. Myers
>Assignee: Doug Cutting
> Fix For: 0.23.0, 0.24.0
>
> Attachments: HDFS-2298.patch, HDFS-2298.patch, HDFS-2298.patch, 
> HDFS-2298.patch, HDFS-2298.patch, HDFS-2298.patch, HDFS-2298.patch
>
>
> The relevant bit of the error:
> {noformat}
> ---
> Test set: org.apache.hadoop.hdfs.TestDfsOverAvroRpc
> ---
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.486 sec <<< 
> FAILURE!
> testWorkingDirectory(org.apache.hadoop.hdfs.TestDfsOverAvroRpc)  Time 
> elapsed: 1.424 sec  <<< ERROR!
> org.apache.avro.AvroTypeException: Two methods with same name: delete
> {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2298) TestDfsOverAvroRpc is failing on trunk

[Doug Cutting (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-2298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137368#comment-13137368
 ] 

Doug Cutting commented on HDFS-2298:


Suresh, yes, I very much agree that this should be merged to 0.23.  However, 
since the reason is to keep the protocols compatible, then we need to first 
merge HADOOP-7524 and HADOOP-7693 to 0.23.  The version of  this patch that 
works without those would still leave 0.23 and trunk incompatible.

> TestDfsOverAvroRpc is failing on trunk
> --
>
> Key: HDFS-2298
> URL: https://issues.apache.org/jira/browse/HDFS-2298
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: test
>Reporter: Aaron T. Myers
>Assignee: Doug Cutting
> Fix For: 0.24.0
>
> Attachments: HDFS-2298.patch, HDFS-2298.patch, HDFS-2298.patch, 
> HDFS-2298.patch, HDFS-2298.patch, HDFS-2298.patch, HDFS-2298.patch
>
>
> The relevant bit of the error:
> {noformat}
> ---
> Test set: org.apache.hadoop.hdfs.TestDfsOverAvroRpc
> ---
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.486 sec <<< 
> FAILURE!
> testWorkingDirectory(org.apache.hadoop.hdfs.TestDfsOverAvroRpc)  Time 
> elapsed: 1.424 sec  <<< ERROR!
> org.apache.avro.AvroTypeException: Two methods with same name: delete
> {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2298) TestDfsOverAvroRpc is failing on trunk

[Doug Cutting (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-2298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13131970#comment-13131970
 ] 

Doug Cutting commented on HDFS-2298:


I believe the plan is to merge this to 0.23 after HADOOP-7524 and then 
HADOOP-7693 have been merged there.  If those issues are not merged to 0.23 
then the best option would be simply to remove this test, as was done in 
HDFS-2383.  Sanjay said he intended merge HADOOP-7524 to 0.23, which gates this.

> TestDfsOverAvroRpc is failing on trunk
> --
>
> Key: HDFS-2298
> URL: https://issues.apache.org/jira/browse/HDFS-2298
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: test
>Reporter: Aaron T. Myers
>Assignee: Doug Cutting
> Fix For: 0.24.0
>
> Attachments: HDFS-2298.patch, HDFS-2298.patch, HDFS-2298.patch, 
> HDFS-2298.patch, HDFS-2298.patch, HDFS-2298.patch, HDFS-2298.patch
>
>
> The relevant bit of the error:
> {noformat}
> ---
> Test set: org.apache.hadoop.hdfs.TestDfsOverAvroRpc
> ---
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.486 sec <<< 
> FAILURE!
> testWorkingDirectory(org.apache.hadoop.hdfs.TestDfsOverAvroRpc)  Time 
> elapsed: 1.424 sec  <<< ERROR!
> org.apache.avro.AvroTypeException: Two methods with same name: delete
> {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2298) TestDfsOverAvroRpc is failing on trunk

[Doug Cutting (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-2298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13117559#comment-13117559
 ] 

Doug Cutting commented on HDFS-2298:


Oops.  I meant HADOOP-7524 above.

> TestDfsOverAvroRpc is failing on trunk
> --
>
> Key: HDFS-2298
> URL: https://issues.apache.org/jira/browse/HDFS-2298
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: test
>Reporter: Aaron T. Myers
>Assignee: Doug Cutting
> Fix For: 0.23.0, 0.24.0
>
> Attachments: HDFS-2298.patch, HDFS-2298.patch, HDFS-2298.patch, 
> HDFS-2298.patch, HDFS-2298.patch, HDFS-2298.patch, HDFS-2298.patch
>
>
> The relevant bit of the error:
> {noformat}
> ---
> Test set: org.apache.hadoop.hdfs.TestDfsOverAvroRpc
> ---
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.486 sec <<< 
> FAILURE!
> testWorkingDirectory(org.apache.hadoop.hdfs.TestDfsOverAvroRpc)  Time 
> elapsed: 1.424 sec  <<< ERROR!
> org.apache.avro.AvroTypeException: Two methods with same name: delete
> {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2298) TestDfsOverAvroRpc is failing on trunk

[Doug Cutting (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-2298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13116910#comment-13116910
 ] 

Doug Cutting commented on HDFS-2298:


> Doug you argued against this.

I still don't like version numbers in methods, but, if semantics are changed in 
a way that cannot be handled otherwise and there's not an obvious better name, 
I don't see an alternative.  Do you?

Nice of you to look for inconsistencies in what I've said, though.  Thanks!

> Given that this has been open for a while, for now can you turn off the test?

I'd prefer to either fix it or remove the code than just disable the test.  If 
there's no will to use Avro RPC in Hadoop then better it should just be 
removed, no?  Sanjay's actively pursuing version interoperability through 
different means, so maybe it's time to remove this stuff.

This has been patch-available, +1'd and passing tests for a few weeks.  Someone 
should have committed it.  Someone like me, I guess, but I was reluctant since 
I've not been that active here lately.  Tests were passing, but have since been 
broken by HADOOP-7524.  I worked today to address that, but if someone instead 
promotes a patch to remove Avro RPC altogether then I'll stop trying to fix 
this.

So, no, I do not intend to turn off the test.  But if you'd like to 'fix' this 
that way, go ahead, I won't block you.


> TestDfsOverAvroRpc is failing on trunk
> --
>
> Key: HDFS-2298
> URL: https://issues.apache.org/jira/browse/HDFS-2298
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: test
>Reporter: Aaron T. Myers
>Assignee: Doug Cutting
> Fix For: 0.23.0, 0.24.0
>
> Attachments: HDFS-2298.patch, HDFS-2298.patch, HDFS-2298.patch, 
> HDFS-2298.patch, HDFS-2298.patch
>
>
> The relevant bit of the error:
> {noformat}
> ---
> Test set: org.apache.hadoop.hdfs.TestDfsOverAvroRpc
> ---
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.486 sec <<< 
> FAILURE!
> testWorkingDirectory(org.apache.hadoop.hdfs.TestDfsOverAvroRpc)  Time 
> elapsed: 1.424 sec  <<< ERROR!
> org.apache.avro.AvroTypeException: Two methods with same name: delete
> {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira