Hongxu Ma created HAWQ-1627: ------------------------------- Summary: Support setting the max protocol message size when talking with HDFS Key: HAWQ-1627 URL: https://issues.apache.org/jira/browse/HAWQ-1627 Project: Apache HAWQ Issue Type: Improvement Components: libhdfs Reporter: Hongxu Ma Assignee: Radar Lei Fix For: 2.4.0.0-incubating
Now, the max size of protocol message in libhdfs is 64MB, and it cannot be adjusted. When the max size is reached (accessing a very big HDFS table/file), will see the following lines in hawq master log: {code} 2018-06-20 11:21:56.768003 CST,,,p75703,th-848100416,,,,0,,,seg-10000,,,,,"LOG","00000","3rd party error log: [libprotobuf ERROR google/protobuf/io/coded_stream.cc:208] A protocol message was rejected because it was too big (more than 67108864 bytes). To increase the limit (or to disable these warnings), see CodedInputStream::SetTotalBytesLimit() in google/protobuf/io/coded_stream.h.",,,,,,,,"SysLoggerMain","syslogger.c",518, 2018-06-20 11:21:56.771657 CST,,,p75703,th-848100416,,,,0,,,seg-10000,,,,,"LOG","00000","3rd party error log: 2018-06-20 11:21:56.771492, p75751, th0x7fffcd7303c0, ERROR Failed to invoke RPC call ""getFsStats"" on server ""localhost:9000"": RpcChannel.cpp: 783: HdfsRpcException: RPC channel to ""localhost:9000"" got protocol mismatch: RPC channel cannot parse response header. @ Hdfs::Internal::RpcChannelImpl::readOneResponse(bool) @ Hdfs::Internal::RpcChannelImpl::checkOneResponse() @ Hdfs::Internal::RpcChannelImpl::invokeInternal(std::__1::shared_ptr<Hdfs::Internal::RpcRemoteCall>) @ Hdfs::Internal::RpcChannelImpl:",,,,,,,,"SysLoggerMain","syslogger.c",518, 2018-06-20 11:21:56.771711 CST,,,p75703,th-848100416,,,,0,,,seg-10000,,,,,"LOG","00000","3rd party error log: :invoke(Hdfs::Internal::RpcCall const&) @ Hdfs::Internal::NamenodeImpl::invoke(Hdfs::Internal::RpcCall const&) @ Hdfs::Internal::NamenodeImpl::getFsStats() @ Hdfs::Internal::NamenodeProxy::getFsStats() @ Hdfs::Internal::FileSystemImpl::getFsStats() @ Hdfs::Internal::FileSystemImpl::connect() @ Hdfs::FileSystem::connect(char const*, char const*, char const*) @ Hdfs::FileSystem::connect(char const*) @ hdfsBuilderConnect @ gpfs_hdfs_connect @ HdfsConnect @ HdfsGetConnection @ HdfsGetFileBlockLocati",,,,,,,,"SysLoggerMain","syslogger.c",518, {code} Considering HDFS has a guc *"ipc.maximum.data.length"* to set it, HAWQ also should has a guc about it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)