[jira] [Commented] (HDDS-2151) Ozone client prints the entire request payload in DEBUG level.
[ https://issues.apache.org/jira/browse/HDDS-2151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933689#comment-16933689 ] Hudson commented on HDDS-2151: -- FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #17336 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/17336/]) HDDS-2151. Ozone client logs the entire request payload at DEBUG level (bharat: rev 1ada99b0bd11d12038f2ed50a0166b5b72d4985e) * (edit) hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/XceiverClientRatis.java > Ozone client prints the entire request payload in DEBUG level. > -- > > Key: HDDS-2151 > URL: https://issues.apache.org/jira/browse/HDDS-2151 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Aravindan Vijayan >Assignee: Doroszlai, Attila >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 1h 10m > Remaining Estimate: 0h > > In XceiverClientRatis.java:221, we have the following snippet where we have a > DEBUG line that prints out the entire Container Request proto. > {code} > ContainerCommandRequestProto finalPayload = > ContainerCommandRequestProto.newBuilder(request) > .setTraceID(TracingUtil.exportCurrentSpan()) > .build(); > boolean isReadOnlyRequest = HddsUtils.isReadOnly(finalPayload); > ByteString byteString = finalPayload.toByteString(); > LOG.debug("sendCommandAsync {} {}", isReadOnlyRequest, finalPayload); > return isReadOnlyRequest ? > getClient().sendReadOnlyAsync(() -> byteString) : > getClient().sendAsync(() -> byteString); > {code} > This causes OOM while writing large (~300MB) keys. > {code} > SLF4J: Failed toString() invocation on an object of type > [org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos$ContainerCommandRequestProto] > Reported exception: > java.lang.OutOfMemoryError: Java heap space > at java.util.Arrays.copyOf(Arrays.java:3332) > at > java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124) > at > java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:649) > at java.lang.StringBuilder.append(StringBuilder.java:202) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormatEscaper.escapeBytes(TextFormatEscaper.java:75) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormatEscaper.escapeBytes(TextFormatEscaper.java:94) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat.escapeBytes(TextFormat.java:1836) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.printFieldValue(TextFormat.java:436) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.printSingleField(TextFormat.java:376) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.printField(TextFormat.java:338) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.print(TextFormat.java:325) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.printFieldValue(TextFormat.java:449) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.printSingleField(TextFormat.java:376) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.printField(TextFormat.java:338) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.print(TextFormat.java:325) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.access$000(TextFormat.java:307) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat.print(TextFormat.java:68) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat.printToString(TextFormat.java:148) > at > org.apache.ratis.thirdparty.com.google.protobuf.AbstractMessage.toString(AbstractMessage.java:117) > at > org.slf4j.helpers.MessageFormatter.safeObjectAppend(MessageFormatter.java:299) > at > org.slf4j.helpers.MessageFormatter.deeplyAppendParameter(MessageFormatter.java:271) > at > org.slf4j.helpers.MessageFormatter.arrayFormat(MessageFormatter.java:233) > at > org.slf4j.helpers.MessageFormatter.arrayFormat(MessageFormatter.java:173) > at org.slf4j.helpers.MessageFormatter.format(MessageFormatter.java:151) > at org.slf4j.impl.Log4jLoggerAdapter.debug(Log4jLoggerAdapter.java:252) > at > org.apache.hadoop.hdds.scm.XceiverClientRatis.sendRequestAsync(XceiverClientRatis.java:221) > at > org.apache.hadoop.hdds.scm.XceiverClientRatis.sendCommandAsync(XceiverClientRatis.java:302) > at >
[jira] [Commented] (HDDS-2151) Ozone client prints the entire request payload in DEBUG level.
[ https://issues.apache.org/jira/browse/HDDS-2151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933599#comment-16933599 ] YiSheng Lien commented on HDDS-2151: Hi [~adoroszlai] Thanks for your PR. You can assign the issue to you:) > Ozone client prints the entire request payload in DEBUG level. > -- > > Key: HDDS-2151 > URL: https://issues.apache.org/jira/browse/HDDS-2151 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Aravindan Vijayan >Assignee: YiSheng Lien >Priority: Major > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > In XceiverClientRatis.java:221, we have the following snippet where we have a > DEBUG line that prints out the entire Container Request proto. > {code} > ContainerCommandRequestProto finalPayload = > ContainerCommandRequestProto.newBuilder(request) > .setTraceID(TracingUtil.exportCurrentSpan()) > .build(); > boolean isReadOnlyRequest = HddsUtils.isReadOnly(finalPayload); > ByteString byteString = finalPayload.toByteString(); > LOG.debug("sendCommandAsync {} {}", isReadOnlyRequest, finalPayload); > return isReadOnlyRequest ? > getClient().sendReadOnlyAsync(() -> byteString) : > getClient().sendAsync(() -> byteString); > {code} > This causes OOM while writing large (~300MB) keys. > {code} > SLF4J: Failed toString() invocation on an object of type > [org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos$ContainerCommandRequestProto] > Reported exception: > java.lang.OutOfMemoryError: Java heap space > at java.util.Arrays.copyOf(Arrays.java:3332) > at > java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124) > at > java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:649) > at java.lang.StringBuilder.append(StringBuilder.java:202) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormatEscaper.escapeBytes(TextFormatEscaper.java:75) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormatEscaper.escapeBytes(TextFormatEscaper.java:94) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat.escapeBytes(TextFormat.java:1836) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.printFieldValue(TextFormat.java:436) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.printSingleField(TextFormat.java:376) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.printField(TextFormat.java:338) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.print(TextFormat.java:325) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.printFieldValue(TextFormat.java:449) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.printSingleField(TextFormat.java:376) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.printField(TextFormat.java:338) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.print(TextFormat.java:325) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.access$000(TextFormat.java:307) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat.print(TextFormat.java:68) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat.printToString(TextFormat.java:148) > at > org.apache.ratis.thirdparty.com.google.protobuf.AbstractMessage.toString(AbstractMessage.java:117) > at > org.slf4j.helpers.MessageFormatter.safeObjectAppend(MessageFormatter.java:299) > at > org.slf4j.helpers.MessageFormatter.deeplyAppendParameter(MessageFormatter.java:271) > at > org.slf4j.helpers.MessageFormatter.arrayFormat(MessageFormatter.java:233) > at > org.slf4j.helpers.MessageFormatter.arrayFormat(MessageFormatter.java:173) > at org.slf4j.helpers.MessageFormatter.format(MessageFormatter.java:151) > at org.slf4j.impl.Log4jLoggerAdapter.debug(Log4jLoggerAdapter.java:252) > at > org.apache.hadoop.hdds.scm.XceiverClientRatis.sendRequestAsync(XceiverClientRatis.java:221) > at > org.apache.hadoop.hdds.scm.XceiverClientRatis.sendCommandAsync(XceiverClientRatis.java:302) > at > org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.writeChunkAsync(ContainerProtocolCalls.java:310) > at > org.apache.hadoop.hdds.scm.storage.BlockOutputStream.writeChunkToContainer(BlockOutputStream.java:601) > at > org.apache.hadoop.hdds.scm.storage.BlockOutputStream.writeChunk(BlockOutputStream.java:459) > at >
[jira] [Commented] (HDDS-2151) Ozone client prints the entire request payload in DEBUG level.
[ https://issues.apache.org/jira/browse/HDDS-2151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933257#comment-16933257 ] Steve Loughran commented on HDDS-2151: -- I think you might want to consider how much of that payload to print at all. Does it ever include secrets? > Ozone client prints the entire request payload in DEBUG level. > -- > > Key: HDDS-2151 > URL: https://issues.apache.org/jira/browse/HDDS-2151 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Aravindan Vijayan >Priority: Major > > In XceiverClientRatis.java:221, we have the following snippet where we have a > DEBUG line that prints out the entire Container Request proto. > {code} > ContainerCommandRequestProto finalPayload = > ContainerCommandRequestProto.newBuilder(request) > .setTraceID(TracingUtil.exportCurrentSpan()) > .build(); > boolean isReadOnlyRequest = HddsUtils.isReadOnly(finalPayload); > ByteString byteString = finalPayload.toByteString(); > LOG.debug("sendCommandAsync {} {}", isReadOnlyRequest, finalPayload); > return isReadOnlyRequest ? > getClient().sendReadOnlyAsync(() -> byteString) : > getClient().sendAsync(() -> byteString); > {code} > This causes OOM while writing large (~300MB) keys. > {code} > SLF4J: Failed toString() invocation on an object of type > [org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos$ContainerCommandRequestProto] > Reported exception: > java.lang.OutOfMemoryError: Java heap space > at java.util.Arrays.copyOf(Arrays.java:3332) > at > java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124) > at > java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:649) > at java.lang.StringBuilder.append(StringBuilder.java:202) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormatEscaper.escapeBytes(TextFormatEscaper.java:75) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormatEscaper.escapeBytes(TextFormatEscaper.java:94) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat.escapeBytes(TextFormat.java:1836) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.printFieldValue(TextFormat.java:436) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.printSingleField(TextFormat.java:376) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.printField(TextFormat.java:338) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.print(TextFormat.java:325) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.printFieldValue(TextFormat.java:449) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.printSingleField(TextFormat.java:376) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.printField(TextFormat.java:338) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.print(TextFormat.java:325) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.access$000(TextFormat.java:307) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat.print(TextFormat.java:68) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat.printToString(TextFormat.java:148) > at > org.apache.ratis.thirdparty.com.google.protobuf.AbstractMessage.toString(AbstractMessage.java:117) > at > org.slf4j.helpers.MessageFormatter.safeObjectAppend(MessageFormatter.java:299) > at > org.slf4j.helpers.MessageFormatter.deeplyAppendParameter(MessageFormatter.java:271) > at > org.slf4j.helpers.MessageFormatter.arrayFormat(MessageFormatter.java:233) > at > org.slf4j.helpers.MessageFormatter.arrayFormat(MessageFormatter.java:173) > at org.slf4j.helpers.MessageFormatter.format(MessageFormatter.java:151) > at org.slf4j.impl.Log4jLoggerAdapter.debug(Log4jLoggerAdapter.java:252) > at > org.apache.hadoop.hdds.scm.XceiverClientRatis.sendRequestAsync(XceiverClientRatis.java:221) > at > org.apache.hadoop.hdds.scm.XceiverClientRatis.sendCommandAsync(XceiverClientRatis.java:302) > at > org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.writeChunkAsync(ContainerProtocolCalls.java:310) > at > org.apache.hadoop.hdds.scm.storage.BlockOutputStream.writeChunkToContainer(BlockOutputStream.java:601) > at > org.apache.hadoop.hdds.scm.storage.BlockOutputStream.writeChunk(BlockOutputStream.java:459) > at > org.apache.hadoop.hdds.scm.storage.BlockOutputStream.write(BlockOutputStream.java:240) > at >
[jira] [Commented] (HDDS-2151) Ozone client prints the entire request payload in DEBUG level.
[ https://issues.apache.org/jira/browse/HDDS-2151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933053#comment-16933053 ] Aravindan Vijayan commented on HDDS-2151: - We have to wrap such DEBUG lines that prints out large payloads within the isDebugEnabled check. > Ozone client prints the entire request payload in DEBUG level. > -- > > Key: HDDS-2151 > URL: https://issues.apache.org/jira/browse/HDDS-2151 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Aravindan Vijayan >Priority: Major > > In XceiverClientRatis.java:221, we have the following snippet where we have a > DEBUG line that prints out the entire Container Request proto. > {code} > ContainerCommandRequestProto finalPayload = > ContainerCommandRequestProto.newBuilder(request) > .setTraceID(TracingUtil.exportCurrentSpan()) > .build(); > boolean isReadOnlyRequest = HddsUtils.isReadOnly(finalPayload); > ByteString byteString = finalPayload.toByteString(); > LOG.debug("sendCommandAsync {} {}", isReadOnlyRequest, finalPayload); > return isReadOnlyRequest ? > getClient().sendReadOnlyAsync(() -> byteString) : > getClient().sendAsync(() -> byteString); > {code} > This causes OOM while writing large (~300MB) keys. > {code} > SLF4J: Failed toString() invocation on an object of type > [org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos$ContainerCommandRequestProto] > Reported exception: > java.lang.OutOfMemoryError: Java heap space > at java.util.Arrays.copyOf(Arrays.java:3332) > at > java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124) > at > java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:649) > at java.lang.StringBuilder.append(StringBuilder.java:202) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormatEscaper.escapeBytes(TextFormatEscaper.java:75) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormatEscaper.escapeBytes(TextFormatEscaper.java:94) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat.escapeBytes(TextFormat.java:1836) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.printFieldValue(TextFormat.java:436) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.printSingleField(TextFormat.java:376) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.printField(TextFormat.java:338) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.print(TextFormat.java:325) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.printFieldValue(TextFormat.java:449) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.printSingleField(TextFormat.java:376) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.printField(TextFormat.java:338) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.print(TextFormat.java:325) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.access$000(TextFormat.java:307) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat.print(TextFormat.java:68) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat.printToString(TextFormat.java:148) > at > org.apache.ratis.thirdparty.com.google.protobuf.AbstractMessage.toString(AbstractMessage.java:117) > at > org.slf4j.helpers.MessageFormatter.safeObjectAppend(MessageFormatter.java:299) > at > org.slf4j.helpers.MessageFormatter.deeplyAppendParameter(MessageFormatter.java:271) > at > org.slf4j.helpers.MessageFormatter.arrayFormat(MessageFormatter.java:233) > at > org.slf4j.helpers.MessageFormatter.arrayFormat(MessageFormatter.java:173) > at org.slf4j.helpers.MessageFormatter.format(MessageFormatter.java:151) > at org.slf4j.impl.Log4jLoggerAdapter.debug(Log4jLoggerAdapter.java:252) > at > org.apache.hadoop.hdds.scm.XceiverClientRatis.sendRequestAsync(XceiverClientRatis.java:221) > at > org.apache.hadoop.hdds.scm.XceiverClientRatis.sendCommandAsync(XceiverClientRatis.java:302) > at > org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.writeChunkAsync(ContainerProtocolCalls.java:310) > at > org.apache.hadoop.hdds.scm.storage.BlockOutputStream.writeChunkToContainer(BlockOutputStream.java:601) > at > org.apache.hadoop.hdds.scm.storage.BlockOutputStream.writeChunk(BlockOutputStream.java:459) > at > org.apache.hadoop.hdds.scm.storage.BlockOutputStream.write(BlockOutputStream.java:240) > at >