[jira] [Commented] (HADOOP-9151) Include RPC error info in RpcResponseHeader instead of sending it separately
[ https://issues.apache.org/jira/browse/HADOOP-9151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13597948#comment-13597948 ] Hudson commented on HADOOP-9151: Integrated in Hadoop-Mapreduce-trunk #1367 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1367/]) HADOOP-9151 Include RPC error info in RpcResponseHeader instead of sending it separately (sanjay Radia) (Revision 1454593) Result = SUCCESS sradia : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1454593 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/dev-support/findbugsExcludeFile.xml * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/proto/RpcHeader.proto > Include RPC error info in RpcResponseHeader instead of sending it separately > > > Key: HADOOP-9151 > URL: https://issues.apache.org/jira/browse/HADOOP-9151 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Sanjay Radia >Assignee: Sanjay Radia > Attachments: HADOOP-9151-2.patch, HADOOP-9151-3.patch, > HADOOP-9151-4.patch, HADOOP-9151-5.patch, HADOOP-9151.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9151) Include RPC error info in RpcResponseHeader instead of sending it separately
[ https://issues.apache.org/jira/browse/HADOOP-9151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13597926#comment-13597926 ] Hudson commented on HADOOP-9151: Integrated in Hadoop-Hdfs-trunk #1339 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1339/]) HADOOP-9151 Include RPC error info in RpcResponseHeader instead of sending it separately (sanjay Radia) (Revision 1454593) Result = SUCCESS sradia : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1454593 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/dev-support/findbugsExcludeFile.xml * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/proto/RpcHeader.proto > Include RPC error info in RpcResponseHeader instead of sending it separately > > > Key: HADOOP-9151 > URL: https://issues.apache.org/jira/browse/HADOOP-9151 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Sanjay Radia >Assignee: Sanjay Radia > Attachments: HADOOP-9151-2.patch, HADOOP-9151-3.patch, > HADOOP-9151-4.patch, HADOOP-9151-5.patch, HADOOP-9151.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9151) Include RPC error info in RpcResponseHeader instead of sending it separately
[ https://issues.apache.org/jira/browse/HADOOP-9151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13597898#comment-13597898 ] Hudson commented on HADOOP-9151: Integrated in Hadoop-Yarn-trunk #150 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/150/]) HADOOP-9151 Include RPC error info in RpcResponseHeader instead of sending it separately (sanjay Radia) (Revision 1454593) Result = SUCCESS sradia : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1454593 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/dev-support/findbugsExcludeFile.xml * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/proto/RpcHeader.proto > Include RPC error info in RpcResponseHeader instead of sending it separately > > > Key: HADOOP-9151 > URL: https://issues.apache.org/jira/browse/HADOOP-9151 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Sanjay Radia >Assignee: Sanjay Radia > Attachments: HADOOP-9151-2.patch, HADOOP-9151-3.patch, > HADOOP-9151-4.patch, HADOOP-9151-5.patch, HADOOP-9151.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9151) Include RPC error info in RpcResponseHeader instead of sending it separately
[ https://issues.apache.org/jira/browse/HADOOP-9151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13597608#comment-13597608 ] Hudson commented on HADOOP-9151: Integrated in Hadoop-trunk-Commit #3444 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3444/]) HADOOP-9151 Include RPC error info in RpcResponseHeader instead of sending it separately (sanjay Radia) (Revision 1454593) Result = SUCCESS sradia : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1454593 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/dev-support/findbugsExcludeFile.xml * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/proto/RpcHeader.proto > Include RPC error info in RpcResponseHeader instead of sending it separately > > > Key: HADOOP-9151 > URL: https://issues.apache.org/jira/browse/HADOOP-9151 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Sanjay Radia >Assignee: Sanjay Radia > Attachments: HADOOP-9151-2.patch, HADOOP-9151-3.patch, > HADOOP-9151-4.patch, HADOOP-9151-5.patch, HADOOP-9151.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9151) Include RPC error info in RpcResponseHeader instead of sending it separately
[ https://issues.apache.org/jira/browse/HADOOP-9151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13596791#comment-13596791 ] Suresh Srinivas commented on HADOOP-9151: - +1 for the patch. > Include RPC error info in RpcResponseHeader instead of sending it separately > > > Key: HADOOP-9151 > URL: https://issues.apache.org/jira/browse/HADOOP-9151 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Sanjay Radia >Assignee: Sanjay Radia > Attachments: HADOOP-9151-2.patch, HADOOP-9151-3.patch, > HADOOP-9151-4.patch, HADOOP-9151-5.patch, HADOOP-9151.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9151) Include RPC error info in RpcResponseHeader instead of sending it separately
[
https://issues.apache.org/jira/browse/HADOOP-9151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13596670#comment-13596670
]
Hadoop QA commented on HADOOP-9151:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12572644/HADOOP-9151-5.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2293//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2293//console
This message is automatically generated.
> Include RPC error info in RpcResponseHeader instead of sending it separately
>
>
> Key: HADOOP-9151
> URL: https://issues.apache.org/jira/browse/HADOOP-9151
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Sanjay Radia
>Assignee: Sanjay Radia
> Attachments: HADOOP-9151-2.patch, HADOOP-9151-3.patch,
> HADOOP-9151-4.patch, HADOOP-9151-5.patch, HADOOP-9151.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9151) Include RPC error info in RpcResponseHeader instead of sending it separately
[
https://issues.apache.org/jira/browse/HADOOP-9151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13596269#comment-13596269
]
Hadoop QA commented on HADOOP-9151:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12572578/HADOOP-9151-4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2291//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2291//console
This message is automatically generated.
> Include RPC error info in RpcResponseHeader instead of sending it separately
>
>
> Key: HADOOP-9151
> URL: https://issues.apache.org/jira/browse/HADOOP-9151
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Sanjay Radia
>Assignee: Sanjay Radia
> Attachments: HADOOP-9151-2.patch, HADOOP-9151-3.patch,
> HADOOP-9151-4.patch, HADOOP-9151.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9151) Include RPC error info in RpcResponseHeader instead of sending it separately
[
https://issues.apache.org/jira/browse/HADOOP-9151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13596186#comment-13596186
]
Hadoop QA commented on HADOOP-9151:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12572571/HADOOP-9151-3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2290//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2290//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-common.html
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2290//console
This message is automatically generated.
> Include RPC error info in RpcResponseHeader instead of sending it separately
>
>
> Key: HADOOP-9151
> URL: https://issues.apache.org/jira/browse/HADOOP-9151
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Sanjay Radia
>Assignee: Sanjay Radia
> Attachments: HADOOP-9151-2.patch, HADOOP-9151-3.patch,
> HADOOP-9151.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9151) Include RPC error info in RpcResponseHeader instead of sending it separately
[
https://issues.apache.org/jira/browse/HADOOP-9151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13595655#comment-13595655
]
Suresh Srinivas commented on HADOOP-9151:
-
+1 for the patch.
Inconsistent synchronization flagged by findbugs is not valid. The warning
needs to be by adding {{in}} field to {{dev-support/findbugsExcludeFile.xml}}
along with the current field {{out}}:
{code:xml}
{code}
> Include RPC error info in RpcResponseHeader instead of sending it separately
>
>
> Key: HADOOP-9151
> URL: https://issues.apache.org/jira/browse/HADOOP-9151
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Sanjay Radia
>Assignee: Sanjay Radia
> Attachments: HADOOP-9151-2.patch, HADOOP-9151.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9151) Include RPC error info in RpcResponseHeader instead of sending it separately
[
https://issues.apache.org/jira/browse/HADOOP-9151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13594305#comment-13594305
]
Hadoop QA commented on HADOOP-9151:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12572235/HADOOP-9151-2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2270//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2270//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-common.html
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2270//console
This message is automatically generated.
> Include RPC error info in RpcResponseHeader instead of sending it separately
>
>
> Key: HADOOP-9151
> URL: https://issues.apache.org/jira/browse/HADOOP-9151
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Sanjay Radia
>Assignee: Sanjay Radia
> Attachments: HADOOP-9151-2.patch, HADOOP-9151.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9151) Include RPC error info in RpcResponseHeader instead of sending it separately
[ https://issues.apache.org/jira/browse/HADOOP-9151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13587672#comment-13587672 ] Todd Lipcon commented on HADOOP-9151: - Sure, I will remove my -1, given that 2.0.3 already broke compatibility (again). Hopefully at some point the rest of the community will come to understand that continuing to break wire compatibility after a ".0" release isn't acceptable. > Include RPC error info in RpcResponseHeader instead of sending it separately > > > Key: HADOOP-9151 > URL: https://issues.apache.org/jira/browse/HADOOP-9151 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Sanjay Radia >Assignee: Sanjay Radia > Attachments: HADOOP-9151.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9151) Include RPC error info in RpcResponseHeader instead of sending it separately
[ https://issues.apache.org/jira/browse/HADOOP-9151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13587574#comment-13587574 ] Suresh Srinivas commented on HADOOP-9151: - Given the discussions from HADOOP-9163 - https://issues.apache.org/jira/browse/HADOOP-9163?focusedCommentId=13581535&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13581535, I think we should move forward with this. Todd and Eli, please confirm. > Include RPC error info in RpcResponseHeader instead of sending it separately > > > Key: HADOOP-9151 > URL: https://issues.apache.org/jira/browse/HADOOP-9151 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Sanjay Radia >Assignee: Sanjay Radia > Attachments: HADOOP-9151.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9151) Include RPC error info in RpcResponseHeader instead of sending it separately
[ https://issues.apache.org/jira/browse/HADOOP-9151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13556801#comment-13556801 ] Luke Lu commented on HADOOP-9151: - Sorry for the noise! I thought that I spotted a logic issue involving sasl that could cause readAndProcess to block indefinitely when a client sends a crafted request. Taught me how not to read code when I should be sleeping :) > Include RPC error info in RpcResponseHeader instead of sending it separately > > > Key: HADOOP-9151 > URL: https://issues.apache.org/jira/browse/HADOOP-9151 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Sanjay Radia >Assignee: Sanjay Radia > Attachments: HADOOP-9151.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9151) Include RPC error info in RpcResponseHeader instead of sending it separately
[ https://issues.apache.org/jira/browse/HADOOP-9151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13556622#comment-13556622 ] Sanjay Radia commented on HADOOP-9151: -- bq. The fact the current protocol is not amenable to nonblocking reader. Can you please add details on the nonblocking reader and how it is related to DOS? > Include RPC error info in RpcResponseHeader instead of sending it separately > > > Key: HADOOP-9151 > URL: https://issues.apache.org/jira/browse/HADOOP-9151 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Sanjay Radia >Assignee: Sanjay Radia > Attachments: HADOOP-9151.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9151) Include RPC error info in RpcResponseHeader instead of sending it separately
[ https://issues.apache.org/jira/browse/HADOOP-9151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13552623#comment-13552623 ] Luke Lu commented on HADOOP-9151: - Binglin's point 3 actually brings up a serious security issue with the current RPC protocol/implementation: it's trivial for any (low-bandwidth) client to DoS any Hadoop RPC service (with unlimited-bandwidth). The fact the current protocol is not amenable to nonblocking reader is a serious issue that needs to be fixed ASAP. I proposed a simple fix in HADOOP-9194. > Include RPC error info in RpcResponseHeader instead of sending it separately > > > Key: HADOOP-9151 > URL: https://issues.apache.org/jira/browse/HADOOP-9151 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Sanjay Radia >Assignee: Sanjay Radia > Attachments: HADOOP-9151.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9151) Include RPC error info in RpcResponseHeader instead of sending it separately
[ https://issues.apache.org/jira/browse/HADOOP-9151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13546316#comment-13546316 ] Eli Collins commented on HADOOP-9151: - Agree. I'm also -1 on making an incompatible change here in the meantime. > Include RPC error info in RpcResponseHeader instead of sending it separately > > > Key: HADOOP-9151 > URL: https://issues.apache.org/jira/browse/HADOOP-9151 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Sanjay Radia >Assignee: Sanjay Radia > Attachments: HADOOP-9151.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9151) Include RPC error info in RpcResponseHeader instead of sending it separately
[ https://issues.apache.org/jira/browse/HADOOP-9151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13546294#comment-13546294 ] Todd Lipcon commented on HADOOP-9151: - What Binglin says makes sense. Let's outline all the changes we want done (or do them in a branch against trunk). We can merge all of these for 3.0 and break compatibility, or if it's obvious that there are big performance improvements available through the changes that can't be done compatibly, we can vote as a community to break compatibility in 2.x. > Include RPC error info in RpcResponseHeader instead of sending it separately > > > Key: HADOOP-9151 > URL: https://issues.apache.org/jira/browse/HADOOP-9151 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Sanjay Radia >Assignee: Sanjay Radia > Attachments: HADOOP-9151.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9151) Include RPC error info in RpcResponseHeader instead of sending it separately
[ https://issues.apache.org/jira/browse/HADOOP-9151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13544625#comment-13544625 ] Binglin Chang commented on HADOOP-9151: --- IMO, current RPC protocol has lot of issues, some can be solved without breaking compatibility, some can not, this JIRA(maybe not worth to change) is just one of them, and many others in HADOOP-8990 are not complete either. How about first propose a complete change solution, with performance test results, then evaluate whether it is worth or not? In addition, list some of the unresolved or potential issues of RPC protocol: # We can't assume msg.writeDelimitedTo avoids buffer copy, I read some protobuf code, in fact it create a huge buffer to do serialization(basically the same as "message.toByteArray()"), unfortunately "writeDelimitedTo" is used everywhere, and compare to buffer allocation, zeroing, some extra buffer copy is hardly performance bottleneck. So some statement in HADOOP-9163 and HADOOP-8084 is not valid. If we truly want a zero copy implementation, perhaps we should reuse CodedOutputStream throughout RPC code rather than each time create a new one, this probably give more performance gains than avoid buffer copy. Whether or not, first some test results is required before change. # Length prefix have no convention at all, we have 4 byte, varint prefixes, and some has -1 to denote special meaning(null string or ping id), some not, all need additional code to deal with. # Protobuf style varint is not convenient for writing aio, non-blocking client/server(still possible), so is -1 prefix, if aio rpc is on the road map, consider unified to 4 byte int prefix with no special meaning. # Merge request/response into single proto has one extra benefits, the whole protocol is now truly compatible, new "sub packets" can be freely added or removed, whether it is a new header or error message, or some new packet type to implement RPC_CONTINUATION_PACKET? > Include RPC error info in RpcResponseHeader instead of sending it separately > > > Key: HADOOP-9151 > URL: https://issues.apache.org/jira/browse/HADOOP-9151 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Sanjay Radia >Assignee: Sanjay Radia > Attachments: HADOOP-9151.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9151) Include RPC error info in RpcResponseHeader instead of sending it separately
[ https://issues.apache.org/jira/browse/HADOOP-9151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13544593#comment-13544593 ] Suresh Srinivas commented on HADOOP-9151: - bq. I think we're making too much out of this issue. It's a small and minor cleanup. Are we? If this is a minor cleanup on an alpha release, the long discussions and invalid veto etc. seems all unnecessary. bq. we explicitly discussed maintaining client wire compatibility after 2.0 ... I think you have missed whole bunch of comments from many folks on this jira. BTW you are also ignoring the alpha part of 2.0.2-alpha. So the release policy makes sense when we drop the alpha monicker. > Include RPC error info in RpcResponseHeader instead of sending it separately > > > Key: HADOOP-9151 > URL: https://issues.apache.org/jira/browse/HADOOP-9151 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Sanjay Radia >Assignee: Sanjay Radia > Attachments: HADOOP-9151.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9151) Include RPC error info in RpcResponseHeader instead of sending it separately
[ https://issues.apache.org/jira/browse/HADOOP-9151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13544360#comment-13544360 ] Eli Collins commented on HADOOP-9151: - I think we're making too much out of this issue. It's a small and minor cleanup. It's also clearly an incompatible change - wire compatibility is just as important as API compatibility. Aside from the fact that this isn't a good trade-off (break wire compatibility for a small cleanup?) we explicitly discussed maintaining client wire compatibility after 2.0 on general@ and our release policy (http://wiki.apache.org/hadoop/Roadmap) covers this: "Thus every release with the same major version is both API and 'on the wire' compatible." I'd be OK with this change for trunk/3.0, even though it would be a bummer if 3.0 clients don't work with the previous major version. > Include RPC error info in RpcResponseHeader instead of sending it separately > > > Key: HADOOP-9151 > URL: https://issues.apache.org/jira/browse/HADOOP-9151 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Sanjay Radia >Assignee: Sanjay Radia > Attachments: HADOOP-9151.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9151) Include RPC error info in RpcResponseHeader instead of sending it separately
[ https://issues.apache.org/jira/browse/HADOOP-9151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13544327#comment-13544327 ] Suresh Srinivas commented on HADOOP-9151: - Todd, just because there are questions about features does not mean the release is being used in production. There are questions about BackupNode as well. I do not think any one is using 2.0.2-alpha in production. Even if they are, I do not think it is a big issue to pick up new libraries. So I do not see this as *hurting* user base. My preference is to do the necessary cleanup and not add any additional code. I do not think CDH or other distribution and their customers should be a concern that needs to be discussed in this jira. > Include RPC error info in RpcResponseHeader instead of sending it separately > > > Key: HADOOP-9151 > URL: https://issues.apache.org/jira/browse/HADOOP-9151 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Sanjay Radia >Assignee: Sanjay Radia > Attachments: HADOOP-9151.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9151) Include RPC error info in RpcResponseHeader instead of sending it separately
[ https://issues.apache.org/jira/browse/HADOOP-9151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13544262#comment-13544262 ] Todd Lipcon commented on HADOOP-9151: - bq. Do you know how many production deployments are there on apache 2.0.2-alpha? Why they will be reluctant to upgrade their entire stack, inspite of having chosen an alpha release and what justifies the expectation of wire compatibility that has never been promised on Apache releases? I don't have a scientific answer to that - haven't done any survey or anything. Non-scientifically, I just searched the hadoop-user list for the last couple of months and I see several questions per week pertaining to 2.x releases. I also see a couple questions per week pertaining to NameNode HA, Federation, or other 2.0-only features. BigTop 0.5.0, which was just released, also bundles 2.0.2-alpha. So even though we've labeled it "alpha", I think it is being treated as de facto stable by a lot of folks who have deployed it. Cloudera hat: Again I don't have exact figures, but a significant portion of our paying customers are running HDFS 2.0.x in production environments. You may claim it's a bad idea because of the labeling, but in practice it has been stable for them and aside from the labeling I think we should treat it as such. Again, if there's a really necessary patch to address something like a security hole, by all means let's break compatibility. But what amounts to a small code cleanup with no advantage to our user base, why hurt them? > Include RPC error info in RpcResponseHeader instead of sending it separately > > > Key: HADOOP-9151 > URL: https://issues.apache.org/jira/browse/HADOOP-9151 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Sanjay Radia >Assignee: Sanjay Radia > Attachments: HADOOP-9151.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9151) Include RPC error info in RpcResponseHeader instead of sending it separately
[ https://issues.apache.org/jira/browse/HADOOP-9151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13544180#comment-13544180 ] Suresh Srinivas commented on HADOOP-9151: - bq. think it's beneficial for Hadoop 2.0.x and 2.1.x to remain wire-compatible with 2.0.2 and earlier I do not think this is such a big issue. I have not seen any reply to the following question I asked earlier. bq. Do you know how many production deployments are there on apache 2.0.2-alpha? Why they will be reluctant to upgrade their entire stack, inspite of having chosen an alpha release and what justifies the expectation of wire compatibility that has never been promised on Apache releases? Lets no add unnecessary complexity to the Apache code. > Include RPC error info in RpcResponseHeader instead of sending it separately > > > Key: HADOOP-9151 > URL: https://issues.apache.org/jira/browse/HADOOP-9151 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Sanjay Radia >Assignee: Sanjay Radia > Attachments: HADOOP-9151.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9151) Include RPC error info in RpcResponseHeader instead of sending it separately
[ https://issues.apache.org/jira/browse/HADOOP-9151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13543642#comment-13543642 ] Todd Lipcon commented on HADOOP-9151: - Hi Sanjay. I mentioned CDH out of transparency, but I think it's beneficial for Hadoop 2.0.x and 2.1.x to remain wire-compatible with 2.0.2 and earlier, if possible. Even outside of CDH, people are using 2.0.x, and we should try to make their lives easier, unless we have a substantial reasoning to break compatibility. As for your question: no, you'd need a client side change in order to be able to interpret the earlier version responses. Certainly we could make a change in CDH which would allow the next version of CDH to be compatible both with Hadoop 2.0.3+ clients and also 2.0.2 (by switching on the IPC version that the client connects with). But, as a community contributor, I think we should strive to maintain wire compatibility between Apache Hadoop versions unless we've got a good reason to break it, vendor distributions aside. > Include RPC error info in RpcResponseHeader instead of sending it separately > > > Key: HADOOP-9151 > URL: https://issues.apache.org/jira/browse/HADOOP-9151 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Sanjay Radia >Assignee: Sanjay Radia > Attachments: HADOOP-9151.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9151) Include RPC error info in RpcResponseHeader instead of sending it separately
[ https://issues.apache.org/jira/browse/HADOOP-9151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13543555#comment-13543555 ] Sanjay Radia commented on HADOOP-9151: -- Todd you have a proposal for making this change compatible with CDH. Can those changes be made in the next dot release of CDH rather than in Apache Hadoop so that CDH is wire compatible with Apache Hadoop? > Include RPC error info in RpcResponseHeader instead of sending it separately > > > Key: HADOOP-9151 > URL: https://issues.apache.org/jira/browse/HADOOP-9151 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Sanjay Radia >Assignee: Sanjay Radia > Attachments: HADOOP-9151.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9151) Include RPC error info in RpcResponseHeader instead of sending it separately
[ https://issues.apache.org/jira/browse/HADOOP-9151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13539206#comment-13539206 ] Arun C Murthy commented on HADOOP-9151: --- bq. For any given issue, I would like to weigh three aspects against each other: [...] If someone finds a big performance improvement or a security bug, and it's difficult to fix it in a compatible way, I'm not going to veto it. Good, we are mostly on same page. However, we need to add a fourth: long-term maintenance cost. If we see a lot of issues such as this, we fix them during the alpha/beta stage *before* we call hadoop-2 as 'stable', rather than maintain all sorts of workarounds for the sake of CDH compatibility. I believe hadoop-2 will be around for a long while and I don't think we should carry that burden - look at how long hadoop-1 has been around. > Include RPC error info in RpcResponseHeader instead of sending it separately > > > Key: HADOOP-9151 > URL: https://issues.apache.org/jira/browse/HADOOP-9151 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Sanjay Radia >Assignee: Sanjay Radia > Attachments: HADOOP-9151.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9151) Include RPC error info in RpcResponseHeader instead of sending it separately
[ https://issues.apache.org/jira/browse/HADOOP-9151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13538914#comment-13538914 ] Suresh Srinivas commented on HADOOP-9151: - Cos and Arun, thank you for your comments. bq. how painful will the change be for users who have already deployed 2.x, whether via CDH, HDP2, or Apache Do you know how many production deployments are there on apache 2.0.2-alpha? Why they will be reluctant to upgrade their entire stack, inspite of having chosen an alpha release and what justifies the expectation of wire compatibility that has never been promised on Apache releases? As regards to HDP, CDH or other distirubtions, I do not think it is a problem that Apache commnity needs to address. Here is some of the thoughts I had written down. Some of it is stale, given others stepping in to the conversation. But still I would like it to be recorded here to avoid wasting energy on these type of discussions in the futures. The technical justifications for this veto is not valid. Some argument I have seen are: h5. This is an incompatible change: I disagree. This changes does not affect API compatibility. No downstream projects or applications need to make any code change. All they have to do is, follow the existing practice of picking up new hadoop jars. This is the only compatibility promised in Hadoop for now. h5. Downstream projects use their own Hadoop client jars and need to be updated: This is a bad idea. Even with Wire/API compatibility, there may be critical fixes in the hadoop libary, needed by downstream projects. Not being able to pick up that with just Hadoop release upgrade, and instead having to deliver these fixes with an upgrade of downstream project is bad. Also I cannot understand picking a copy of 2.0-alpha library prematurely and not waiting until GA. h5. A non Apache distribution is tagged stable. Hence no incompatible change can be made in Apache: I cannot understand how a non Apache distribution can be tagged stable when underlying Apache code does not make any such promise. What a non Apache distribution does, how they tag their release, what content they include is outside the control of Apache community. Hence, while as a community we may make it easy for other distributions, a veto that mandates Apache community * must * play nice to an outside distribution is without merit. BTW I fail to understand why it is such a big deal. You can chose not to include this and related changes in CDH. h5. Apache 2.0 HDFS and common are stable: They may be. But the release still carries alpha tag. Because an individual thinks it is stable, does not mean every one should treat it like a GA/stable release. If the tag currently we are using does not reflect the reality, lets change it based on community decision. Only way to indicate stability is, Apache release changes the stability tag and not based on individual perception. h5. The benefit of this change does not warrant incompatible change: First, see my argument that about why this is not incompatible. Second, citing the returns are not worth it, is not a technical argument. In fact you have said and so do others that this is a cleanup that is necessary and should have been done. At alpha stage of the release, this kind of change should go in. As Arun has stated, that is the main reason for alpha tag. The fact that it could make another distro incompatible when it goes in, is not a problem that we should discuss/solve here. h5. Only few individuals in the world get the benefit of this change: It does not matter whether 100s or a very few get benefit out of a change. It improves the quality and completes some of the cleanups that were happening. I can turn this argument of limited people benefit around and say, only a very few installations of 2.0.2-alpha (how many?) and those users know what to expect from an alpha release. Based on all this, I believe this veto makes no sense in the context of Apache hadoop releases. Wearing my Apache hat, it is not justified to expect Apache community to pay for decisions made elsewhere. These are some ways you can go around the issue: # Change your distribution to include a layer that can support compatibility with Apache releases. # Upgrade all the components when the next release comes out or include these changes at your own convenience, later. # Pick this change up in a dot release, say 4.1 where you could upgrade the entire stack. # There could be other alternatives, but I am not here to solve issues outside the scope of Apache Hadoop releases. > Include RPC error info in RpcResponseHeader instead of sending it separately > > > Key: HADOOP-9151 > URL: https://issues.apache.org/jira/browse/HADOOP-9151 > Proj
[jira] [Commented] (HADOOP-9151) Include RPC error info in RpcResponseHeader instead of sending it separately
[ https://issues.apache.org/jira/browse/HADOOP-9151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13538879#comment-13538879 ] Todd Lipcon commented on HADOOP-9151: - bq. Also, it isn't true that the discussion was limited to YARN, see the link I posted above w.r.t HDFS from Suresh: http://s.apache.org/qQp. I'm pretty sure you've ascribed to above views too (w.r.t changing wire-protocols on HDFS etc.) - here is your own view that HDFS protocols aren't 'baked' yet: http://s.apache.org/PRh Two things about my post there -- for one, it was 8 months ago. More people have adopted 2.0 since then, and I think wire compatibility grows more important as more people adopt the system. Second, I did say "+1 to locking down client wire protocol compatibility". This proposed patch would break client compatibility. bq. However, we might have to do more changes of this nature in the future (maybe major bugs or new features etc. before hadoop-2 hits 'stable') and I really hope you won't be vetoing further development on grounds of 'CDH compatibility'. For any given issue, I would like to weigh three aspects against each other: 1) how much effort would it be to do compatibly? 2) how painful will the change be for users who have already deployed 2.x, whether via CDH, HDP2, or Apache), and 3) how much value is in the change. If someone finds a big performance improvement or a security bug, and it's difficult to fix it in a compatible way, I'm not going to veto it. But if it's just 'cleanup' with no advantage to users, then I'm of the opinion that the negatives are stronger than the positives. > Include RPC error info in RpcResponseHeader instead of sending it separately > > > Key: HADOOP-9151 > URL: https://issues.apache.org/jira/browse/HADOOP-9151 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Sanjay Radia >Assignee: Sanjay Radia > Attachments: HADOOP-9151.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9151) Include RPC error info in RpcResponseHeader instead of sending it separately
[ https://issues.apache.org/jira/browse/HADOOP-9151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13538753#comment-13538753 ] Arun C Murthy commented on HADOOP-9151: --- Thanks, I appreciate you being upfront - but it doesn't change the merits of your veto. As others have pointed out, you are welcome to continue shipping CDH without these changes. The 'alpha'/'beta' tags are meant to be a warning to end-users about the stability of the software (apis, wire-protocols etc.) and users can make their own decisions. We've discussed this many times for very good reasons - for e.g. the PB-based HDFS protocols are very raw (~12 months old at this point v/s Writables' RPC viz. nearly 4-5yrs now) and it's not surprising we hit these issues, there may be further down the road. Also, it isn't true that the discussion was limited to YARN, see the link I posted above w.r.t HDFS from Suresh: http://s.apache.org/qQp. I'm pretty sure you've ascribed to above views too (w.r.t changing wire-protocols on HDFS etc.) - here is your own view that HDFS protocols aren't 'baked' yet: http://s.apache.org/PRh Having said that, I agree we maybe overblowing the current issue. However, we might have to do more changes of this nature in the future (maybe major bugs or new features etc. before hadoop-2 hits 'stable') and I really hope you won't be vetoing further development on grounds of 'CDH compatibility'. > Include RPC error info in RpcResponseHeader instead of sending it separately > > > Key: HADOOP-9151 > URL: https://issues.apache.org/jira/browse/HADOOP-9151 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Sanjay Radia >Assignee: Sanjay Radia > Attachments: HADOOP-9151.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9151) Include RPC error info in RpcResponseHeader instead of sending it separately
[ https://issues.apache.org/jira/browse/HADOOP-9151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13538637#comment-13538637 ] Todd Lipcon commented on HADOOP-9151: - I tried to be transparent and honest by explaining my perspective both from the employer hat and from the Apache hat. I know that, had I not mentioned the employer, folks would have suspected my second motivation anyway, so I stick with "honesty is the best policy." My understanding is that the Apache process is designed such that committers are certainly allowed to represent the interests of their employers. Most of us are here because our employers pay us to work on the project, so it's only to be expected that we make technical decisions in the projects based on what our employers want. Regardless of the employer, there are people using the Apache 2.0.x releases in real scenarios, and it's going to be painful for them to break IPC compatibility as well. Sure, we're _allowed_ to by the "alpha" naming. But just because we are allowed to do something doesn't mean we _should_ if it represents pain for downstream consumers. As for the threads you pointed out, most of the discussion there was regarding YARN/MR API changes. I agree that those projects are very much in flux and have fewer people depending on them. The changes being referenced also seem to be much more major and important things. In contrast, the change proposed here is a very small detail which won't affect further development -- the IPC code is touched fairly rarely, and has a very straightforward function. I don't see the proposed improvement making any substantial difference in future projects in this area. I'm happy to do the patch which allows making this change compatibly if the issue is finding the time to do the more complicated change. > Include RPC error info in RpcResponseHeader instead of sending it separately > > > Key: HADOOP-9151 > URL: https://issues.apache.org/jira/browse/HADOOP-9151 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Sanjay Radia >Assignee: Sanjay Radia > Attachments: HADOOP-9151.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9151) Include RPC error info in RpcResponseHeader instead of sending it separately
[ https://issues.apache.org/jira/browse/HADOOP-9151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13538570#comment-13538570 ] Arun C Murthy commented on HADOOP-9151: --- Todd - I don't see how you can veto a patch based on commercial interests of your employer. It's invalid, you can't hold the community hostage to your commercial interests. This isn't a one-off either, this is exactly the reason we've had the 'alpha' moniker - iron out early issues in protocols/apis etc. in hadoop-2. Incidentally, this is exactly why YARN-142 & MAPREDUCE-4067 are being done *before* we even call hadoop-2 as 'beta'. We've also discussed this publicly - see http://s.apache.org/28V, http://s.apache.org/qQp & http://s.apache.org/xLs. > Include RPC error info in RpcResponseHeader instead of sending it separately > > > Key: HADOOP-9151 > URL: https://issues.apache.org/jira/browse/HADOOP-9151 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Sanjay Radia >Assignee: Sanjay Radia > Attachments: HADOOP-9151.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9151) Include RPC error info in RpcResponseHeader instead of sending it separately
[ https://issues.apache.org/jira/browse/HADOOP-9151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13537734#comment-13537734 ] Konstantin Boudnik commented on HADOOP-9151: "Wearing my hat of application developer on top of Hadoop" I'd rather see the wrinkles ironed out as early as possible. The reason is: it is enough of PITA to deal with all the changes in serialization layer in the matter of transit to HDFS2. Getting back to it in 3.x when an incompatible changes will be introduced again - I don't think this is the message Hadoop community needs to be sending around. "Wearing my Apache hat": it won't be real fair to punish Hadoop consumers, just because Cloudera was too eager to come ahead with the CDH version based on an "alpha" version of ASF Hadoop. > Include RPC error info in RpcResponseHeader instead of sending it separately > > > Key: HADOOP-9151 > URL: https://issues.apache.org/jira/browse/HADOOP-9151 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Sanjay Radia >Assignee: Sanjay Radia > Attachments: HADOOP-9151.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9151) Include RPC error info in RpcResponseHeader instead of sending it separately
[ https://issues.apache.org/jira/browse/HADOOP-9151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13537676#comment-13537676 ] Binglin Chang commented on HADOOP-9151: --- bq. Not sure I understand this comment. Isn't the current byte[] coming from a protobuf message already? I mean for every rpc, there are already many new Proto objects created(C++ can reuse proto objects, but java can't...), the overhead may already way larger than buffer copies, so avoiding buffer copy may not help much. bq. Can you please explain how does the current proposal prevent this? There are 2 problems: 1. As you know, protobuf library doesn't provide non-blocking style method for reading varint length prefix, in order to handle rpc in a non-blocking style, I have to write my own version of non-blocking reading varint, in this scenario, 4 byte fix length prefix are more suitable for non-blocking io. 2. With every length prefix packet, I need 2 more callbacks to read length and read packet body, if each rpc call only have one request packet and one response packet, the code have much less callbacks. Since its already a fact, I just want it keep length prefix as few as possible. As I already finish the rpc code using the complex code, honestly I'm all OK with keeping compatibility or breaking it. But if we do want to break it, please consider the proposal more throughly, consider potential usage, make it high quality protocol. @Todd When I write async rpc client, The prefixed string(WritableUtils.readString) make -1(or MAX_UINT) to represent null string, this is a little annoying, introducing some special code to handle writable string. > Include RPC error info in RpcResponseHeader instead of sending it separately > > > Key: HADOOP-9151 > URL: https://issues.apache.org/jira/browse/HADOOP-9151 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Sanjay Radia >Assignee: Sanjay Radia > Attachments: HADOOP-9151.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9151) Include RPC error info in RpcResponseHeader instead of sending it separately
[ https://issues.apache.org/jira/browse/HADOOP-9151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13537625#comment-13537625 ] Todd Lipcon commented on HADOOP-9151: - Well, how do we resolve it? The two options I see are: 1) We follow the compatibility path I outlined above in a comment, or: 2) Since I've vetoed the patch for branch-2, bring it to the larger development community to discuss and perhaps vote. > Include RPC error info in RpcResponseHeader instead of sending it separately > > > Key: HADOOP-9151 > URL: https://issues.apache.org/jira/browse/HADOOP-9151 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Sanjay Radia >Assignee: Sanjay Radia > Attachments: HADOOP-9151.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9151) Include RPC error info in RpcResponseHeader instead of sending it separately
[ https://issues.apache.org/jira/browse/HADOOP-9151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13537623#comment-13537623 ] Suresh Srinivas commented on HADOOP-9151: - bq. I won't debate it's a wart. It's just not worth the pain to fix it in branch-2 when it doesn't buy us any real advantages. I have to disagree. I also want to make sure we avoid this kind of discussions and -1s for incompatible changes, just because a distribution decided to call it stable. We avoid incompatible changes when Apache release makes it stable. bq. It's a 5kb patch...? Obviously not on this patch. On cleaning RPC layer. > Include RPC error info in RpcResponseHeader instead of sending it separately > > > Key: HADOOP-9151 > URL: https://issues.apache.org/jira/browse/HADOOP-9151 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Sanjay Radia >Assignee: Sanjay Radia > Attachments: HADOOP-9151.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9151) Include RPC error info in RpcResponseHeader instead of sending it separately
[ https://issues.apache.org/jira/browse/HADOOP-9151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13537620#comment-13537620 ] Todd Lipcon commented on HADOOP-9151: - bq. getting rid of writable in serialized rpc request and response is equally important It's not a custom writable. It's a 4-byte length-prefixed string (which happens to be implemented by a class called WritableUtils). This is super easy to serialize in any language. In fact, anyone implementing RPC already needs to implement length-prefixed strings in order to send the protos themselves. I won't debate it's a wart. It's just not worth the pain to fix it in branch-2 when it doesn't buy us any real advantages. bq. I and Sanjay have spent quite a lot of time on this It's a 5kb patch...? > Include RPC error info in RpcResponseHeader instead of sending it separately > > > Key: HADOOP-9151 > URL: https://issues.apache.org/jira/browse/HADOOP-9151 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Sanjay Radia >Assignee: Sanjay Radia > Attachments: HADOOP-9151.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9151) Include RPC error info in RpcResponseHeader instead of sending it separately
[ https://issues.apache.org/jira/browse/HADOOP-9151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13537613#comment-13537613 ] Suresh Srinivas commented on HADOOP-9151: - bq. So, given that I already consider HDFS to be "stable", and know people running this branch in production scenarios where a rolling upgrade is required, I would make the same argument that we should not break compatibility. You may consider it to be "stable". But the release is called 2.0.2-"alpha". Before calling it GA, I was planning to run through a checklist to make sure our wire compatibility story is rock solid. That is what is happening some of the jiras that are being filed now. bq. If this issue were a case of a big performance improvement, or a security issue, or even a feature improvement, I would weigh it stronger While these are compelling reasons for you, getting rid of writable in serialized rpc request and response is equally important. I and Sanjay have spent quite a lot of time on this. Seeing it completed before GA is a good goal. > Include RPC error info in RpcResponseHeader instead of sending it separately > > > Key: HADOOP-9151 > URL: https://issues.apache.org/jira/browse/HADOOP-9151 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Sanjay Radia >Assignee: Sanjay Radia > Attachments: HADOOP-9151.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9151) Include RPC error info in RpcResponseHeader instead of sending it separately
[
https://issues.apache.org/jira/browse/HADOOP-9151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13537565#comment-13537565
]
Todd Lipcon commented on HADOOP-9151:
-
{quote}
We are talking about Apache 2.0.x-alpha release here. How CDH manages its
distribution, backward compatibility does not guide how Apache releases are
done or what goes into Apache releases. The fact that you chose include a
content that is not in trunk or decided to tag a release in some other way
should not put constraints on the Apache releases. Wearing my Apache hat on, if
this is the main reason for -1, then it has no merit.
If there was no issue to CDH distribution, would you have objected to this
change?
I would like others to comment on why a vendor's distribution or compatibility
to it should put artificial constraints in Apache.
{quote}
Regardless of the existence of CDH, I would have argued that HDFS and Common
should have been labeled "Stable" months ago. For people who don't care to run
MR2, I've found HDFS2 to be far more stable than HDFS1, in addition to offering
many other benefits. But we've already beaten that horse to death on the
mailing list.
So, given that I already consider HDFS to be "stable", and know people running
this branch in production scenarios where a rolling upgrade is required, I
would make the same argument that we should not break compatibility.
bq. I would like others to comment on why a vendor's distribution or
compatibility to it should put artificial constraints in Apache.
There are lots of people out there using this distro. If we break
compatibility, people will have a harder time moving from the distro _back_ to
Apache, if that's the angle you want to look at it from.
If this issue were a case of a big performance improvement, or a security
issue, or even a feature improvement, I would weigh it stronger. But given that
it's a code cleanup that brings no improvement at all to users, and only a very
slight improvement for the 3 or 4 people in the world who are trying to
implement Hadoop-compatible RPC in another langauge (one of whom happens to be
me!), I think it needs much more motivation.
> Include RPC error info in RpcResponseHeader instead of sending it separately
>
>
> Key: HADOOP-9151
> URL: https://issues.apache.org/jira/browse/HADOOP-9151
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Sanjay Radia
>Assignee: Sanjay Radia
> Attachments: HADOOP-9151.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9151) Include RPC error info in RpcResponseHeader instead of sending it separately
[ https://issues.apache.org/jira/browse/HADOOP-9151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13537518#comment-13537518 ] Suresh Srinivas commented on HADOOP-9151: - bq. As far as I know we have not broken RPC or API compatibility at all since 2.0.0, and I would be against any case where we do (not just this one). We have not broken API compatibility in a long time. Even though RPC compatibility is not broken, there have been many changes marked as incompatible (I counted 6 of them). Just because incompatible changes have not been done in RPC does not mean it cannot be done. bq. As for the labeling of alpha, I have been arguing against calling it alpha for several months The reason of having alpha tag is so that we do not have to provide the stricter guarantees of a GA release. I am glad that we have retained it so far, so that these kind of changes can happen. bq. In the spirit of full disclosure: wearing my Cloudera hat, we have a distribution based on the Hadoop 2 code line. We are not going to break wire compatibility within minor updates of this distribution. So, if branch-2 breaks compatibility, then our distro will become incompatible with branch-2, which is no good. We are talking about Apache 2.0.x-alpha release here. How CDH manages its distribution, backward compatibility does not guide how Apache releases are done or what goes into Apache releases. The fact that you chose include a content that is not in trunk or decided to tag a release in some other way should not put constraints on the Apache releases. Wearing my Apache hat on, if this is the main reason for -1, then it has no merit. If there was no issue to CDH distribution, would you have objected to this change? I would like others to comment on why a vendor's distribution or compatibility to it should put artificial constraints in Apache. > Include RPC error info in RpcResponseHeader instead of sending it separately > > > Key: HADOOP-9151 > URL: https://issues.apache.org/jira/browse/HADOOP-9151 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Sanjay Radia >Assignee: Sanjay Radia > Attachments: HADOOP-9151.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9151) Include RPC error info in RpcResponseHeader instead of sending it separately
[ https://issues.apache.org/jira/browse/HADOOP-9151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13537477#comment-13537477 ] Todd Lipcon commented on HADOOP-9151: - bq. Not sure if I follow. When HBase is installed on top a Hadoop cluster doesn't it pickup the hadoop jars from the installed hadoop? Not always. If you download the HBase tarball, for example, and build it with the "Hadoop 2" profile, it will end up with its own copies of the hadoop jars. bq. Any plans to check this into Hadoop? This would be useful to the wider community and we can speed up libhdfs. The client is just a simple RPC client - not a full HDFS client, and I didn't get it to a really usable "production quality" state - just enough to do a "listFiles" on a running NN. I'll try to throw it up on github at some point. My point, though, was just to say that, even though this area of the protocol is slightly messy, it's by far not the biggest obstacle to writing foreign language implementations of RPC. bq. I understand that it could be messy for downstream projects. But that is what an alpha release tag is meant to indicate. BTW since 2.0.0-alpha we have had many incompatible changes as well As far as I know we have not broken RPC or API compatibility at all since 2.0.0, and I would be against any case where we do (not just this one). As for the labeling of alpha, I have been arguing against calling it alpha for several months, but there's no clear bylaws on how the labeling changes, etc. So, given that I can't change the labeling, I'll just represent my feelings on individual code changes - and this is one that I can't legitimately support on branch-2. In the spirit of full disclosure: wearing my Cloudera hat, we have a distribution based on the Hadoop 2 code line. We are not going to break wire compatibility within minor updates of this distribution. So, if branch-2 breaks compatibility, then our distro will become incompatible with branch-2, which is no good. Wearing my Apache hat: it would be nice if Cloudera, Hortonworks, Apache, and anyone else distributing "Hadoop 2" can remain fully wire- and API-compatible. If this change goes in, it will serve to fracture the ecosystem more, and make it harder for the community to move freely between different versions. I imagine other distributors would much prefer that all distros are wire-compatible, since it makes it easier, for example, for one of our customers to go and work with another vendor without rebuilding all their code with a new client jar (one of the main points of value in open source!) So, to be clear, -1 on this change for branch-2 unless there is a compatibility path. > Include RPC error info in RpcResponseHeader instead of sending it separately > > > Key: HADOOP-9151 > URL: https://issues.apache.org/jira/browse/HADOOP-9151 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Sanjay Radia >Assignee: Sanjay Radia > Attachments: HADOOP-9151.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9151) Include RPC error info in RpcResponseHeader instead of sending it separately
[ https://issues.apache.org/jira/browse/HADOOP-9151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13537337#comment-13537337 ] Sanjay Radia commented on HADOOP-9151: -- bq. @todd I've implemented a simple client for it in C++ Any plans to check this into Hadoop? This would be useful to the wider community and we can speed up libhdfs. > Include RPC error info in RpcResponseHeader instead of sending it separately > > > Key: HADOOP-9151 > URL: https://issues.apache.org/jira/browse/HADOOP-9151 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Sanjay Radia >Assignee: Sanjay Radia > Attachments: HADOOP-9151.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9151) Include RPC error info in RpcResponseHeader instead of sending it separately
[ https://issues.apache.org/jira/browse/HADOOP-9151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13537330#comment-13537330 ] Sanjay Radia commented on HADOOP-9151: -- @todd .. HBase issue Not sure if I follow. When HBase is installed on top a Hadoop cluster doesn't it pickup the hadoop jars from the installed hadoop? > Include RPC error info in RpcResponseHeader instead of sending it separately > > > Key: HADOOP-9151 > URL: https://issues.apache.org/jira/browse/HADOOP-9151 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Sanjay Radia >Assignee: Sanjay Radia > Attachments: HADOOP-9151.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9151) Include RPC error info in RpcResponseHeader instead of sending it separately
[ https://issues.apache.org/jira/browse/HADOOP-9151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13537323#comment-13537323 ] Suresh Srinivas commented on HADOOP-9151: - bq. I'd be +1 on this for 3.0 but not for branch-2, unless you can figure out some way of maintaining a compatibility path. The main reason why we started the protobuf work was to make sure once we declare 2.0 as GA we will do the needed complex code to maintain the compatibility. So breaking compatibility for 3.0 should not be necessary. I understand that it could be messy for downstream projects. But that is what an alpha release tag is meant to indicate. BTW since 2.0.0-alpha we have had many incompatible changes as well. Granted, they may not affect HBase, but does affect other folks some way or the other. BTW I have reflected same sentiments in the discussion thread of making 2.0.0 beta as well. If we are going to make these kinds of changes even harder to make with the beta tag, I am reluctant to change the tag. > Include RPC error info in RpcResponseHeader instead of sending it separately > > > Key: HADOOP-9151 > URL: https://issues.apache.org/jira/browse/HADOOP-9151 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Sanjay Radia >Assignee: Sanjay Radia > Attachments: HADOOP-9151.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9151) Include RPC error info in RpcResponseHeader instead of sending it separately
[ https://issues.apache.org/jira/browse/HADOOP-9151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13537307#comment-13537307 ] Todd Lipcon commented on HADOOP-9151: - bq. I agree that if possible we should avoid breaking the compatibility. But when a cleanup can make the implementation lot more clear, I prefer making that change. That is the reason why we use alpha and any one who uses that release should be ready to adopt the changes. The implementation is slightly more clear, I agree. We should have done it this way to begin with. But IMO it's really not worth it. There are now downstream projects building against Hadoop-2 releases such as HBase. If we break compat between 2.0.2 and a later release, then HBase users will have the messy "make sure you replace your HDFS client jar inside HBase's lib directory with the exact right version or else get weird error messages" nonsense come back again. I'd be +1 on this for 3.0 but not for branch-2, unless you can figure out some way of maintaining a compatibility path. For example, we could do something like this in 2.0: - add an optional flag to the request protobuf indicating that the new style exception response is supported - mark the embedded exception - new server checks the flag: if the flag is there, it sends the new-style (in-PB) exception response. If it is not there, it uses the old style response for compatibility - when the new client gets an ERROR response, but exceptionClassName is not found in the response PB, then it knows it is an old style server, and reads the writables following the response (like the old protocol) This would be compatible as follows: - old client -> old server: obviously compatible - old client -> new server: doesn't send flag, so new server responds with old response - new client -> old server: sends flag, but the old server ignores it (since it's part of a PB, this is automatic). server responds with old style response. - new client -> new server: sends flag, so new server responds with new style response. Obviously, all this makes the code more complex. So, I think it would be better to just leave branch-2 alone, and make the improvement in branch-3/trunk. > Include RPC error info in RpcResponseHeader instead of sending it separately > > > Key: HADOOP-9151 > URL: https://issues.apache.org/jira/browse/HADOOP-9151 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Sanjay Radia >Assignee: Sanjay Radia > Attachments: HADOOP-9151.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9151) Include RPC error info in RpcResponseHeader instead of sending it separately
[ https://issues.apache.org/jira/browse/HADOOP-9151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13537284#comment-13537284 ] Suresh Srinivas commented on HADOOP-9151: - Minor nit - For better readability you may want to use conditional statement using ?: for setting exception class name and stack trace. > Include RPC error info in RpcResponseHeader instead of sending it separately > > > Key: HADOOP-9151 > URL: https://issues.apache.org/jira/browse/HADOOP-9151 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Sanjay Radia >Assignee: Sanjay Radia > Attachments: HADOOP-9151.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9151) Include RPC error info in RpcResponseHeader instead of sending it separately
[ https://issues.apache.org/jira/browse/HADOOP-9151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13537225#comment-13537225 ] Suresh Srinivas commented on HADOOP-9151: - bq. I don't think it's a good idea to break compatibility for what amounts to a surface-level cleanup... bq. Despite the "alpha" label, there are a lot of people using it, and breaking the compat should require a really good reason. I agree that if possible we should avoid breaking the compatibility. But when a cleanup can make the implementation lot more clear, I prefer making that change. That is the reason why we use alpha and any one who uses that release should be ready to adopt the changes. bq. Plus that the overhead of creating new proto objects for every rpc may already take a lot more resource than buffer copy Not sure I understand this comment. Isn't the current byte[] coming from a protobuf message already? bq. And another concern is this protocol may be bad for async non-blocking io Can you please explain how does the current proposal prevent this? > Include RPC error info in RpcResponseHeader instead of sending it separately > > > Key: HADOOP-9151 > URL: https://issues.apache.org/jira/browse/HADOOP-9151 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Sanjay Radia >Assignee: Sanjay Radia > Attachments: HADOOP-9151.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9151) Include RPC error info in RpcResponseHeader instead of sending it separately
[ https://issues.apache.org/jira/browse/HADOOP-9151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13536806#comment-13536806 ] Binglin Chang commented on HADOOP-9151: --- Yes, it makes sense to avoid buffer copy. But I suspect the performance penalty of buffer copy isn't that much(rpc call should be small in most cases), but the penalty of complex code logic may be more. Plus that the overhead of creating new proto objects for every rpc may already take a lot more resource than buffer copy, so buffer copy may be not the bottleneck at all. And another concern is this protocol may be bad for async non-blocking io, which just want a total length, receive enough info to process, then call rpc handles. More distinct packats make more handlers/callbacks. > Include RPC error info in RpcResponseHeader instead of sending it separately > > > Key: HADOOP-9151 > URL: https://issues.apache.org/jira/browse/HADOOP-9151 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Sanjay Radia >Assignee: Sanjay Radia > Attachments: HADOOP-9151.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9151) Include RPC error info in RpcResponseHeader instead of sending it separately
[ https://issues.apache.org/jira/browse/HADOOP-9151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13536380#comment-13536380 ] Sanjay Radia commented on HADOOP-9151: -- Binglin, you are suggesting to merge the rpc-request body inside the RpcRequestHeader and the rpc-repsponse body inside RpcResponseHeader. You are right in that we could add a bytes field to each of the headers and that it would not break the Multiple rpc engine abstraction. Generally it is better to keep the distinct "messages" distinct, each length-deliminated. The Protobuf rpc messages of the various RPC services get generated independently anyway. As far as efficiency there are two separate issues. # wire packets - we are writing data into a tcp stream via a bufferOutputStream and so there should be no difference. # internal buffers - indeed putting the body inside the header can cause extra copying depending on how you write your code. A further note on internal data copying: our rpc code should be changed to use CodeOutputStream to minimize data copying; I will file a jira for that. While I was changing the code, I noticed that the ProtobufRpcEngine.proto includes the rpc message inside the header and indeed that will force extra copying and in my opinion should have been kept distinct. > Include RPC error info in RpcResponseHeader instead of sending it separately > > > Key: HADOOP-9151 > URL: https://issues.apache.org/jira/browse/HADOOP-9151 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Sanjay Radia >Assignee: Sanjay Radia > Attachments: HADOOP-9151.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9151) Include RPC error info in RpcResponseHeader instead of sending it separately
[ https://issues.apache.org/jira/browse/HADOOP-9151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13535196#comment-13535196 ] Todd Lipcon commented on HADOOP-9151: - bq. Todd, not sure what you mean here. From what I know 0.23.x does not have any of the protobuf changes or all the other changes that have happened in ipc right? Sorry, my mistake. Still, 2.x has been out for quite some time now, and I don't think it's a good idea to break compatibility for what amounts to a surface-level cleanup. Despite the "alpha" label, there are a lot of people using it, and breaking the compat should require a really good reason. Certainly it's a shame to have some non-PB items in there, but they're very simple serializations of strings. I've implemented a simple client for it in C++ and it wasn't at all difficult to deal with the few places where writables snuck into the header. > Include RPC error info in RpcResponseHeader instead of sending it separately > > > Key: HADOOP-9151 > URL: https://issues.apache.org/jira/browse/HADOOP-9151 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Sanjay Radia >Assignee: Sanjay Radia > Attachments: HADOOP-9151.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9151) Include RPC error info in RpcResponseHeader instead of sending it separately
[ https://issues.apache.org/jira/browse/HADOOP-9151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13534663#comment-13534663 ] Binglin Chang commented on HADOOP-9151: --- What are the concerns of not merge rpc response and request body(as no type bytes, so still support multiple rpc engine) with RpcRequestHeader and RpcResponseHeader? So there is only one request/response packet per rpc call, easier and potentially more efficient? > Include RPC error info in RpcResponseHeader instead of sending it separately > > > Key: HADOOP-9151 > URL: https://issues.apache.org/jira/browse/HADOOP-9151 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Sanjay Radia >Assignee: Sanjay Radia > Attachments: HADOOP-9151.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9151) Include RPC error info in RpcResponseHeader instead of sending it separately
[
https://issues.apache.org/jira/browse/HADOOP-9151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13534600#comment-13534600
]
Hadoop QA commented on HADOOP-9151:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12561388/HADOOP-9151.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common:
org.apache.hadoop.ha.TestZKFailoverController
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/1903//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HADOOP-Build/1903//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-common.html
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/1903//console
This message is automatically generated.
> Include RPC error info in RpcResponseHeader instead of sending it separately
>
>
> Key: HADOOP-9151
> URL: https://issues.apache.org/jira/browse/HADOOP-9151
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Sanjay Radia
>Assignee: Sanjay Radia
> Attachments: HADOOP-9151.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9151) Include RPC error info in RpcResponseHeader instead of sending it separately
[ https://issues.apache.org/jira/browse/HADOOP-9151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13534581#comment-13534581 ] Suresh Srinivas commented on HADOOP-9151: - bq. We currently have client compatibility between 0.23.x and 2.x as far as I know Todd, not sure what you mean here. From what I know 0.23.x does not have any of the protobuf changes or all the other changes that have happened in ipc right? > Include RPC error info in RpcResponseHeader instead of sending it separately > > > Key: HADOOP-9151 > URL: https://issues.apache.org/jira/browse/HADOOP-9151 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Sanjay Radia >Assignee: Sanjay Radia > Attachments: HADOOP-9151.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9151) Include RPC error info in RpcResponseHeader instead of sending it separately
[ https://issues.apache.org/jira/browse/HADOOP-9151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13534493#comment-13534493 ] Sanjay Radia commented on HADOOP-9151: -- On RPC error, the server side sends the ExceptionClassName and StackTrace after the RpcResponseHeader as UTF-8 strings. These two should be included in the RpcResponseHeader so that the rpc protocol is fully in protobuf. I had meant to do this but forgot. > Include RPC error info in RpcResponseHeader instead of sending it separately > > > Key: HADOOP-9151 > URL: https://issues.apache.org/jira/browse/HADOOP-9151 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Sanjay Radia >Assignee: Sanjay Radia > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9151) Include RPC error info in RpcResponseHeader instead of sending it separately
[ https://issues.apache.org/jira/browse/HADOOP-9151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13534490#comment-13534490 ] Todd Lipcon commented on HADOOP-9151: - I don't think it's a good idea to make incompatible changes at this point. We currently have client compatibility between 0.23.x and 2.x as far as I know, and this would break it without a strong motivation. > Include RPC error info in RpcResponseHeader instead of sending it separately > > > Key: HADOOP-9151 > URL: https://issues.apache.org/jira/browse/HADOOP-9151 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Sanjay Radia >Assignee: Sanjay Radia > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
