[
https://issues.apache.org/jira/browse/KUDU-3450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17690045#comment-17690045
]
ASF subversion and git services commented on KUDU-3450:
-------------------------------------------------------
Commit ae22d32ef5895a02a763665cfd2812aa61be5ab3 in kudu's branch
refs/heads/master from Alexey Serbin
[ https://gitbox.apache.org/repos/asf?p=kudu.git;h=ae22d32ef ]
KUDU-3450 handling of oversized messages in subprocess server
This patch adds a new flag --subprocess_max_message_size_bytes to
configure the maximum allowed size for the body of the response message
received by the subprocess server, with default value of 8 MiB.
Prior to this patch, the hard-coded limit was 1 MiB.
I also updated the default values for the following flags, increasing
them 4x times:
* --subprocess_request_queue_size_bytes (4 --> 16 MiB)
* --subprocess_response_queue_size_bytes (4 --> 16 MiB)
With this patch, the behavior of the subprocess server has changed
when encountering an error while reading a response message from
its subprocess. Now, the server does not just bail upon a larger than
expected message or in case of other error, but rather tries to read
out and discard the message to clear the communication channel,
so there is a chance to receive next messages from the subprocess.
A new metric has been added for the subprocess server to report on
the number of read and discarded messages because of various errors
while reading and decoding the data from the communication channel:
* server_dropped_messages
In addition, this patch introduces a few test scenarios to cover the
newly introduced functionality.
I also sprayed the relevant code with syntactic sugar of structural
binding since the code requires C++17-capable compiler anyways.
A follow-up patch should take care of the corresponding client-side
components that are used to run the Ranger client as a subprocess.
Change-Id: I05b09e757f304b22e37438c2445ecc161ef412c9
Reviewed-on: http://gerrit.cloudera.org:8080/19500
Reviewed-by: Attila Bukor <abu...@apache.org>
Tested-by: Alexey Serbin <ale...@apache.org>
> SubprocessProtocol has a hard-coded limit on message size, but RangerClient
> doesn't honor that while generating requests
> ------------------------------------------------------------------------------------------------------------------------
>
> Key: KUDU-3450
> URL: https://issues.apache.org/jira/browse/KUDU-3450
> Project: Kudu
> Issue Type: Bug
> Components: subprocess
> Affects Versions: 1.12.0, 1.13.0, 1.14.0, 1.15.0, 1.16.0
> Reporter: Alexey Serbin
> Priority: Major
>
> As one can see from the code in the
> [SubprocessProtocol|https://github.com/apache/kudu/blob/9684200713b5e1bf258437527127bd98acfa1e42/src/kudu/subprocess/subprocess_protocol.cc#L106-L110]
> and in [MessageIO utility Java
> class|https://github.com/apache/kudu/blob/9684200713b5e1bf258437527127bd98acfa1e42/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageIO.java#L69-L73],
> it's not possible to receive messages in PB format that are larger than a
> pre-defined limit. Also, there isn't a way to pass
> {{SubprocessRequestPB::request}} and {{SubprocessResponsePB::response}} data
> in multiple chunks in the SubprocessProtocol by design.
> The default setting for the maximum message size is 1MiByte in the
> implementation of [the relevant
> C++|https://github.com/apache/kudu/blob/9684200713b5e1bf258437527127bd98acfa1e42/src/kudu/subprocess/subprocess_protocol.cc#L46]
> and [Java subprocess
> components|https://github.com/apache/kudu/blob/9684200713b5e1bf258437527127bd98acfa1e42/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessConfiguration.java#L54].
> The implementation of the related C++ components has the setting effectively
> hard-coded and there isn't a way to change it. The implementation of the
> related Java components has this setting configurable via the {{maxMsgBytes}}
> configuration property.
> Anyways, nether the code in {{RangerClient}} nor the code in
> {{RangerProtocolHandler}} honors the limit:
> *
> [RangerClient::AuthorizeActionMultipleColumns()|https://github.com/apache/kudu/blob/9684200713b5e1bf258437527127bd98acfa1e42/src/kudu/ranger/ranger_client.cc#L435-L448]
> *
> [RangerClient::AuthorizeActionMultipleTables()|https://github.com/apache/kudu/blob/9684200713b5e1bf258437527127bd98acfa1e42/src/kudu/ranger/ranger_client.cc#L468-L492]
> *
> [ProtocolHandler.unpackAndExecuteRequest()|https://github.com/apache/kudu/blob/9684200713b5e1bf258437527127bd98acfa1e42/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/ProtocolHandler.java#L47-L56]
> With that, it's possible to get into an impasse situation from the both
> sides. For example, if a large request has been generated by
> {{RangerClient::AuthorizeActionMultipleColumns()}} or
> {{RangerClient::AuthorizeActionMultipleTables()}}, the following stack trace
> might be observed in the output of the subprocess that runs the Ranger client
> plugin:
> {noformat}
> Exception in thread "main" java.util.concurrent.CompletionException:
> org.apache.kudu.subprocess.KuduSubprocessException: Unable to read the
> protobuf message
> at
> java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273)
> at
> java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280)
> at
> java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1629)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.kudu.subprocess.KuduSubprocessException: Unable to read
> the protobuf message
> at org.apache.kudu.subprocess.MessageReader.run(MessageReader.java:74)
> at
> java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626)
> ... 3 more
> Caused by: java.io.IOException: message size (2763197) exceeds maximum
> message size (1048576)
> at org.apache.kudu.subprocess.MessageIO.readBytes(MessageIO.java:71)
> at org.apache.kudu.subprocess.MessageReader.run(MessageReader.java:68)
> ... 4 more
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
