[
https://issues.apache.org/jira/browse/IMPALA-13020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17847420#comment-17847420
]
ASF subversion and git services commented on IMPALA-13020:
----------------------------------------------------------
Commit bcff4df6194b2f192d937bb9c031721feccb69df in impala's branch
refs/heads/master from Joe McDonnell
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=bcff4df61 ]
IMPALA-13020 (part 2): Split out external vs internal Thrift max message size
The Thrift max message size is designed to protect against malicious
messages that consume a lot of memory on the receiver. This is an
important security measure for externally facing services, but it
can interfere with internal communication within the cluster.
Currently, the max message size is controlled by a single startup
flag for both. This puts tensions between having a low value to
protect against malicious messages versus having a high value to
avoid issues with internal communication (e.g. large statestore
updates).
This introduces a new flag thrift_external_rpc_max_message_size to
specify the limit for externally-facing services. The current
thrift_rpc_max_message_size now applies only for internal services.
Splitting them apart allows setting a much higher value for
internal services (64GB) while leaving the externally facing services
using the current 2GB limit.
This modifies various code locations that wrap a Thrift transport to
pass in the original transport's TConfiguration. This also adds DCHECKs
to make sure that the new transport inherits the max message size. This
limits the locations where we actually need to set max message size.
ThriftServer/ThriftServerBuilder have a setting "is_external_facing"
which can be specified on each ThriftServer. This modifies statestore
and catalog to set is_external_facing to false. All other servers stay
with the default of true.
Testing:
- This adds a test case to verify that is_external_facing uses the
higher limit.
- Ran through the steps in testdata/scale_test_metadata/README.md
and updated the value in that doc.
- Created many tables to push the catalog-update topic to be >2GB
and verified that statestore successfully sends it when an impalad
restarts.
Change-Id: Ib9a649ef49a8a99c7bd9a1b73c37c4c621661311
Reviewed-on: http://gerrit.cloudera.org:8080/21420
Tested-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
Reviewed-by: Riza Suminto <riza.sumi...@cloudera.com>
Reviewed-by: Michael Smith <michael.sm...@cloudera.com>
> catalog-topic updates >2GB do not work due to Thrift's max message size
> -----------------------------------------------------------------------
>
> Key: IMPALA-13020
> URL: https://issues.apache.org/jira/browse/IMPALA-13020
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 4.2.0, Impala 4.3.0
> Reporter: Joe McDonnell
> Priority: Critical
>
> Thrift 0.16.0 added a max message size to protect against malicious packets
> that can consume a large amount of memory on the receiver side. This max
> message size is a signed 32-bit integer, so it maxes out at 2GB (which we set
> via thrift_rpc_max_message_size).
> In catalog v1, the catalog-update statestore topic can become larger than 2GB
> when there are a large number of tables / partitions / files. If this happens
> and an Impala coordinator needs to start up (or needs a full topic update for
> any other reason), it is expecting the statestore to send it the full topic
> update, but the coordinator actually can't process the message. The
> deserialization of the message hits the 2GB max message size limit and fails.
> On the statestore side, it shows this message:
> {noformat}
> I0418 16:54:51.727290 3844140 statestore.cc:507] Preparing initial
> catalog-update topic update for
> impa...@mcdonnellthrift.vpc.cloudera.com:27000. Size = 2.27 GB
> I0418 16:54:53.889446 3844140 thrift-util.cc:198] TSocket::write_partial()
> send() <Host: mcdonnellthrift.vpc.cloudera.com Port: 23000>: Broken pipe
> I0418 16:54:53.889488 3844140 client-cache.cc:82] ReopenClient(): re-creating
> client for mcdonnellthrift.vpc.cloudera.com:23000
> I0418 16:54:53.889493 3844140 thrift-util.cc:198] TSocket::write_partial()
> send() <Host: mcdonnellthrift.vpc.cloudera.com Port: 23000>: Broken pipe
> I0418 16:54:53.889503 3844140 thrift-client.cc:116] Error closing connection
> to: mcdonnellthrift.vpc.cloudera.com:23000, ignoring (write() send(): Broken
> pipe)
> I0418 16:54:56.052882 3844140 thrift-util.cc:198] TSocket::write_partial()
> send() <Host: mcdonnellthrift.vpc.cloudera.com Port: 23000>: Broken pipe
> I0418 16:54:56.052932 3844140 client-cache.h:363] RPC Error: Client for
> mcdonnellthrift.vpc.cloudera.com:23000 hit an unexpected exception: write()
> send(): Broken pipe, type: N6apache6thrift9transport19TTransportExceptionE,
> rpc: N6impala20TUpdateStateResponseE, send: not done
> I0418 16:54:56.052937 3844140 client-cache.cc:174] Broken Connection, destroy
> client for mcdonnellthrift.vpc.cloudera.com:23000{noformat}
> On the Impala side, it doesn't give a good error, but we see this:
> {noformat}
> I0418 16:54:53.889683 3214537 TAcceptQueueServer.cpp:355] New connection to
> server StatestoreSubscriber from client <Host: 127.0.0.1 Port: 49632>
> I0418 16:54:54.080694 3214136 Frontend.java:1837] Waiting for local catalog
> to be initialized, attempt: 110
> I0418 16:54:56.080920 3214136 Frontend.java:1837] Waiting for local catalog
> to be initialized, attempt: 111
> I0418 16:54:58.081131 3214136 Frontend.java:1837] Waiting for local catalog
> to be initialized, attempt: 112
> I0418 16:55:00.081358 3214136 Frontend.java:1837] Waiting for local catalog
> to be initialized, attempt: 113{noformat}
> With a patch Thrift that allows an int64_t max message size and setting that
> to a larger value, the Impala was able to start up (even without restarting
> the statestored).
> Some clusters that upgrade to a newer version may hit this, as Thrift didn't
> use to enforce this limit, so this is something we should fix to avoid
> upgrade issues.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
