[ https://issues.apache.org/jira/browse/HAWQ-1627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hongxu Ma reassigned HAWQ-1627: ------------------------------- Assignee: Hongxu Ma (was: Radar Lei) > Support setting the max protocol message size when talking with HDFS > -------------------------------------------------------------------- > > Key: HAWQ-1627 > URL: https://issues.apache.org/jira/browse/HAWQ-1627 > Project: Apache HAWQ > Issue Type: Improvement > Components: libhdfs > Reporter: Hongxu Ma > Assignee: Hongxu Ma > Priority: Major > Fix For: 2.4.0.0-incubating > > > Now, the max size of protocol message in libhdfs is 64MB, and it cannot be > adjusted. > When the max size is reached (accessing a very big HDFS table/file), will see > the following lines in hawq master log: > {code} > 2018-06-20 11:21:56.768003 > CST,,,p75703,th-848100416,,,,0,,,seg-10000,,,,,"LOG","00000","3rd party error > log: > [libprotobuf ERROR google/protobuf/io/coded_stream.cc:208] A protocol message > was rejected because it was too big (more than 67108864 bytes). To increase > the limit (or to > disable these warnings), see CodedInputStream::SetTotalBytesLimit() in > google/protobuf/io/coded_stream.h.",,,,,,,,"SysLoggerMain","syslogger.c",518, > 2018-06-20 11:21:56.771657 > CST,,,p75703,th-848100416,,,,0,,,seg-10000,,,,,"LOG","00000","3rd party error > log: > 2018-06-20 11:21:56.771492, p75751, th0x7fffcd7303c0, ERROR Failed to invoke > RPC call ""getFsStats"" on server ""localhost:9000"": > RpcChannel.cpp: 783: HdfsRpcException: RPC channel to ""localhost:9000"" got > protocol mismatch: RPC channel cannot parse response header. > @ Hdfs::Internal::RpcChannelImpl::readOneResponse(bool) > @ Hdfs::Internal::RpcChannelImpl::checkOneResponse() > @ > Hdfs::Internal::RpcChannelImpl::invokeInternal(std::__1::shared_ptr<Hdfs::Internal::RpcRemoteCall>) > @ > Hdfs::Internal::RpcChannelImpl:",,,,,,,,"SysLoggerMain","syslogger.c",518, > 2018-06-20 11:21:56.771711 > CST,,,p75703,th-848100416,,,,0,,,seg-10000,,,,,"LOG","00000","3rd party error > log: > :invoke(Hdfs::Internal::RpcCall const&) > @ Hdfs::Internal::NamenodeImpl::invoke(Hdfs::Internal::RpcCall const&) > @ Hdfs::Internal::NamenodeImpl::getFsStats() > @ Hdfs::Internal::NamenodeProxy::getFsStats() > @ Hdfs::Internal::FileSystemImpl::getFsStats() > @ Hdfs::Internal::FileSystemImpl::connect() > @ Hdfs::FileSystem::connect(char const*, char const*, char const*) > @ Hdfs::FileSystem::connect(char const*) > @ hdfsBuilderConnect > @ gpfs_hdfs_connect > @ HdfsConnect > @ HdfsGetConnection > @ HdfsGetFileBlockLocati",,,,,,,,"SysLoggerMain","syslogger.c",518, > {code} > Considering HDFS has a guc *"ipc.maximum.data.length"* to set it, HAWQ also > should has a guc about it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)