[jira] [Commented] (HADOOP-18143) toString method of RpcCall throws IllegalArgumentException
[
https://issues.apache.org/jira/browse/HADOOP-18143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17497588#comment-17497588
]
András Győri commented on HADOOP-18143:
---
Created a PR for it, in which I wrap the affected functions with a mutex. As
getRequestHeader is more like getOrCreateRequestHeader, I did not see the
benefit of dividing the lock as separate read/write locks (as in both methods
we would just end up using the write lock anyway). Not sure whether it is
acceptable in terms of performance.
cc [~vinayakumarb]
> toString method of RpcCall throws IllegalArgumentException
> --
>
> Key: HADOOP-18143
> URL: https://issues.apache.org/jira/browse/HADOOP-18143
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: András Győri
>Assignee: András Győri
>Priority: Critical
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> We have observed breaking tests such as TestApplicationACLs. We have located
> the root cause, which is HADOOP-18082. It seems that there is a concurrency
> issue within ProtobufRpcEngine2. When using a debugger, the missing fields
> are there, hence the suspicion of concurrency problem. The stack trace:
> {noformat}
> java.lang.IllegalArgumentException
> at java.nio.Buffer.position(Buffer.java:244)
> at
> org.apache.hadoop.ipc.RpcWritable$ProtobufWrapper.readFrom(RpcWritable.java:131)
> at org.apache.hadoop.ipc.RpcWritable$Buffer.getValue(RpcWritable.java:232)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine2$RpcProtobufRequest.getRequestHeader(ProtobufRpcEngine2.java:645)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine2$RpcProtobufRequest.toString(ProtobufRpcEngine2.java:663)
> at java.lang.String.valueOf(String.java:3425)
> at java.lang.StringBuilder.append(StringBuilder.java:516)
> at org.apache.hadoop.ipc.Server$RpcCall.toString(Server.java:1328)
> at java.lang.String.valueOf(String.java:3425)
> at java.lang.StringBuilder.append(StringBuilder.java:516)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:3097){noformat}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18143) toString method of RpcCall throws IllegalArgumentException
[
https://issues.apache.org/jira/browse/HADOOP-18143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17499036#comment-17499036
]
Ayush Saxena commented on HADOOP-18143:
---
[~quapaw] Trying to understand the issue.
Seems the concurrency issue tends to occur when {{requestHeader}} is in
progress of being written, means half written and during that phase someone
calls {{getRequestHeader}} or say {{toString}} which indirectly calls
{{{}getRequestHeader{}}}. Can happen while debugging since the debugger calls
{{{}toString{}}}, or if someone is logging when the write is in progress.
May be making both {{getRequestHeader}} and {{writeTo}} synchronized or having
a lock as in the PR should solve this problem.
Seems this problem should surface even for ProtobufRpcEngine as well. I don't
see any reason which will prevent it there. If my understanding is correct.
I think this would surface during debugging only, if we remove the log
statement, right? I am not sure of the performance penalty what making a method
synchronized or taking lock can bring. But just a simple search on the internet
shows it would bring in some, if that is true, I don't think we can take that
cost, considering this is some commonly used code.
> toString method of RpcCall throws IllegalArgumentException
> --
>
> Key: HADOOP-18143
> URL: https://issues.apache.org/jira/browse/HADOOP-18143
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: András Győri
>Assignee: András Győri
>Priority: Critical
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> We have observed breaking tests such as TestApplicationACLs. We have located
> the root cause, which is HADOOP-18082. It seems that there is a concurrency
> issue within ProtobufRpcEngine2. When using a debugger, the missing fields
> are there, hence the suspicion of concurrency problem. The stack trace:
> {noformat}
> java.lang.IllegalArgumentException
> at java.nio.Buffer.position(Buffer.java:244)
> at
> org.apache.hadoop.ipc.RpcWritable$ProtobufWrapper.readFrom(RpcWritable.java:131)
> at org.apache.hadoop.ipc.RpcWritable$Buffer.getValue(RpcWritable.java:232)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine2$RpcProtobufRequest.getRequestHeader(ProtobufRpcEngine2.java:645)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine2$RpcProtobufRequest.toString(ProtobufRpcEngine2.java:663)
> at java.lang.String.valueOf(String.java:3425)
> at java.lang.StringBuilder.append(StringBuilder.java:516)
> at org.apache.hadoop.ipc.Server$RpcCall.toString(Server.java:1328)
> at java.lang.String.valueOf(String.java:3425)
> at java.lang.StringBuilder.append(StringBuilder.java:516)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:3097){noformat}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18143) toString method of RpcCall throws IllegalArgumentException
[
https://issues.apache.org/jira/browse/HADOOP-18143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17499056#comment-17499056
]
András Győri commented on HADOOP-18143:
---
[~ayushtkn] Locking does involve some overhead, that is why we wanted some
secondary opinion on this. Synchronization is a coarser locking mechanism, I
think reentrant lock is better in this case. However, I do think that toString
is used outside of debugging, because things started to fall apart when I have
changed something in it (namely to return empty string when requestHeader is
null). All in all, this concurrency issue could occur any time, even in
production, affecting writeTo and toString as well. It seems, that
ProtobufRpcEngine is affected as well.
> toString method of RpcCall throws IllegalArgumentException
> --
>
> Key: HADOOP-18143
> URL: https://issues.apache.org/jira/browse/HADOOP-18143
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: András Győri
>Assignee: András Győri
>Priority: Critical
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> We have observed breaking tests such as TestApplicationACLs. We have located
> the root cause, which is HADOOP-18082. It seems that there is a concurrency
> issue within ProtobufRpcEngine2. When using a debugger, the missing fields
> are there, hence the suspicion of concurrency problem. The stack trace:
> {noformat}
> java.lang.IllegalArgumentException
> at java.nio.Buffer.position(Buffer.java:244)
> at
> org.apache.hadoop.ipc.RpcWritable$ProtobufWrapper.readFrom(RpcWritable.java:131)
> at org.apache.hadoop.ipc.RpcWritable$Buffer.getValue(RpcWritable.java:232)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine2$RpcProtobufRequest.getRequestHeader(ProtobufRpcEngine2.java:645)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine2$RpcProtobufRequest.toString(ProtobufRpcEngine2.java:663)
> at java.lang.String.valueOf(String.java:3425)
> at java.lang.StringBuilder.append(StringBuilder.java:516)
> at org.apache.hadoop.ipc.Server$RpcCall.toString(Server.java:1328)
> at java.lang.String.valueOf(String.java:3425)
> at java.lang.StringBuilder.append(StringBuilder.java:516)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:3097){noformat}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18143) toString method of RpcCall throws IllegalArgumentException
[
https://issues.apache.org/jira/browse/HADOOP-18143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17499112#comment-17499112
]
Ayush Saxena commented on HADOOP-18143:
---
Did a git blame on ProtobufRpcEngine Class from where the code was extracted.
Considering this exist there as well
[~kihwal] / [~daryn] in case you folks have any feedback or pointers here...
> toString method of RpcCall throws IllegalArgumentException
> --
>
> Key: HADOOP-18143
> URL: https://issues.apache.org/jira/browse/HADOOP-18143
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: András Győri
>Assignee: András Győri
>Priority: Critical
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> We have observed breaking tests such as TestApplicationACLs. We have located
> the root cause, which is HADOOP-18082. It seems that there is a concurrency
> issue within ProtobufRpcEngine2. When using a debugger, the missing fields
> are there, hence the suspicion of concurrency problem. The stack trace:
> {noformat}
> java.lang.IllegalArgumentException
> at java.nio.Buffer.position(Buffer.java:244)
> at
> org.apache.hadoop.ipc.RpcWritable$ProtobufWrapper.readFrom(RpcWritable.java:131)
> at org.apache.hadoop.ipc.RpcWritable$Buffer.getValue(RpcWritable.java:232)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine2$RpcProtobufRequest.getRequestHeader(ProtobufRpcEngine2.java:645)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine2$RpcProtobufRequest.toString(ProtobufRpcEngine2.java:663)
> at java.lang.String.valueOf(String.java:3425)
> at java.lang.StringBuilder.append(StringBuilder.java:516)
> at org.apache.hadoop.ipc.Server$RpcCall.toString(Server.java:1328)
> at java.lang.String.valueOf(String.java:3425)
> at java.lang.StringBuilder.append(StringBuilder.java:516)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:3097){noformat}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
