[ https://issues.apache.org/jira/browse/HDDS-2151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16933257#comment-16933257 ]
Steve Loughran commented on HDDS-2151: -------------------------------------- I think you might want to consider how much of that payload to print at all. Does it ever include secrets? > Ozone client prints the entire request payload in DEBUG level. > -------------------------------------------------------------- > > Key: HDDS-2151 > URL: https://issues.apache.org/jira/browse/HDDS-2151 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Reporter: Aravindan Vijayan > Priority: Major > > In XceiverClientRatis.java:221, we have the following snippet where we have a > DEBUG line that prints out the entire Container Request proto. > {code} > ContainerCommandRequestProto finalPayload = > ContainerCommandRequestProto.newBuilder(request) > .setTraceID(TracingUtil.exportCurrentSpan()) > .build(); > boolean isReadOnlyRequest = HddsUtils.isReadOnly(finalPayload); > ByteString byteString = finalPayload.toByteString(); > LOG.debug("sendCommandAsync {} {}", isReadOnlyRequest, finalPayload); > return isReadOnlyRequest ? > getClient().sendReadOnlyAsync(() -> byteString) : > getClient().sendAsync(() -> byteString); > {code} > This causes OOM while writing large (~300MB) keys. > {code} > SLF4J: Failed toString() invocation on an object of type > [org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos$ContainerCommandRequestProto] > Reported exception: > java.lang.OutOfMemoryError: Java heap space > at java.util.Arrays.copyOf(Arrays.java:3332) > at > java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124) > at > java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:649) > at java.lang.StringBuilder.append(StringBuilder.java:202) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormatEscaper.escapeBytes(TextFormatEscaper.java:75) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormatEscaper.escapeBytes(TextFormatEscaper.java:94) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat.escapeBytes(TextFormat.java:1836) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.printFieldValue(TextFormat.java:436) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.printSingleField(TextFormat.java:376) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.printField(TextFormat.java:338) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.print(TextFormat.java:325) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.printFieldValue(TextFormat.java:449) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.printSingleField(TextFormat.java:376) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.printField(TextFormat.java:338) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.print(TextFormat.java:325) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.access$000(TextFormat.java:307) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat.print(TextFormat.java:68) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat.printToString(TextFormat.java:148) > at > org.apache.ratis.thirdparty.com.google.protobuf.AbstractMessage.toString(AbstractMessage.java:117) > at > org.slf4j.helpers.MessageFormatter.safeObjectAppend(MessageFormatter.java:299) > at > org.slf4j.helpers.MessageFormatter.deeplyAppendParameter(MessageFormatter.java:271) > at > org.slf4j.helpers.MessageFormatter.arrayFormat(MessageFormatter.java:233) > at > org.slf4j.helpers.MessageFormatter.arrayFormat(MessageFormatter.java:173) > at org.slf4j.helpers.MessageFormatter.format(MessageFormatter.java:151) > at org.slf4j.impl.Log4jLoggerAdapter.debug(Log4jLoggerAdapter.java:252) > at > org.apache.hadoop.hdds.scm.XceiverClientRatis.sendRequestAsync(XceiverClientRatis.java:221) > at > org.apache.hadoop.hdds.scm.XceiverClientRatis.sendCommandAsync(XceiverClientRatis.java:302) > at > org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.writeChunkAsync(ContainerProtocolCalls.java:310) > at > org.apache.hadoop.hdds.scm.storage.BlockOutputStream.writeChunkToContainer(BlockOutputStream.java:601) > at > org.apache.hadoop.hdds.scm.storage.BlockOutputStream.writeChunk(BlockOutputStream.java:459) > at > org.apache.hadoop.hdds.scm.storage.BlockOutputStream.write(BlockOutputStream.java:240) > at > org.apache.hadoop.ozone.client.io.BlockOutputStreamEntry.write(BlockOutputStreamEntry.java:129) > SLF4J: Failed toString() invocation on an object of type > [org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos$ContainerCommandRequestProto] > Reported exception: > java.lang.OutOfMemoryError: Java heap space > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org