[jira] [Updated] (HDDS-2151) Ozone client prints the entire request payload in DEBUG level.
[ https://issues.apache.org/jira/browse/HDDS-2151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDDS-2151: - Fix Version/s: 0.5.0 Resolution: Fixed Status: Resolved (was: Patch Available) > Ozone client prints the entire request payload in DEBUG level. > -- > > Key: HDDS-2151 > URL: https://issues.apache.org/jira/browse/HDDS-2151 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Aravindan Vijayan >Assignee: Doroszlai, Attila >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 1h > Remaining Estimate: 0h > > In XceiverClientRatis.java:221, we have the following snippet where we have a > DEBUG line that prints out the entire Container Request proto. > {code} > ContainerCommandRequestProto finalPayload = > ContainerCommandRequestProto.newBuilder(request) > .setTraceID(TracingUtil.exportCurrentSpan()) > .build(); > boolean isReadOnlyRequest = HddsUtils.isReadOnly(finalPayload); > ByteString byteString = finalPayload.toByteString(); > LOG.debug("sendCommandAsync {} {}", isReadOnlyRequest, finalPayload); > return isReadOnlyRequest ? > getClient().sendReadOnlyAsync(() -> byteString) : > getClient().sendAsync(() -> byteString); > {code} > This causes OOM while writing large (~300MB) keys. > {code} > SLF4J: Failed toString() invocation on an object of type > [org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos$ContainerCommandRequestProto] > Reported exception: > java.lang.OutOfMemoryError: Java heap space > at java.util.Arrays.copyOf(Arrays.java:3332) > at > java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124) > at > java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:649) > at java.lang.StringBuilder.append(StringBuilder.java:202) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormatEscaper.escapeBytes(TextFormatEscaper.java:75) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormatEscaper.escapeBytes(TextFormatEscaper.java:94) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat.escapeBytes(TextFormat.java:1836) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.printFieldValue(TextFormat.java:436) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.printSingleField(TextFormat.java:376) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.printField(TextFormat.java:338) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.print(TextFormat.java:325) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.printFieldValue(TextFormat.java:449) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.printSingleField(TextFormat.java:376) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.printField(TextFormat.java:338) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.print(TextFormat.java:325) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.access$000(TextFormat.java:307) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat.print(TextFormat.java:68) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat.printToString(TextFormat.java:148) > at > org.apache.ratis.thirdparty.com.google.protobuf.AbstractMessage.toString(AbstractMessage.java:117) > at > org.slf4j.helpers.MessageFormatter.safeObjectAppend(MessageFormatter.java:299) > at > org.slf4j.helpers.MessageFormatter.deeplyAppendParameter(MessageFormatter.java:271) > at > org.slf4j.helpers.MessageFormatter.arrayFormat(MessageFormatter.java:233) > at > org.slf4j.helpers.MessageFormatter.arrayFormat(MessageFormatter.java:173) > at org.slf4j.helpers.MessageFormatter.format(MessageFormatter.java:151) > at org.slf4j.impl.Log4jLoggerAdapter.debug(Log4jLoggerAdapter.java:252) > at > org.apache.hadoop.hdds.scm.XceiverClientRatis.sendRequestAsync(XceiverClientRatis.java:221) > at > org.apache.hadoop.hdds.scm.XceiverClientRatis.sendCommandAsync(XceiverClientRatis.java:302) > at > org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.writeChunkAsync(ContainerProtocolCalls.java:310) > at > org.apache.hadoop.hdds.scm.storage.BlockOutputStream.writeChunkToContainer(BlockOutputStream.java:601) > at > org.apache.hadoop.hdds.scm.storage.BlockOutputStream.writeChunk(BlockOutputStream.java:459) > at >
[jira] [Updated] (HDDS-2151) Ozone client prints the entire request payload in DEBUG level.
[ https://issues.apache.org/jira/browse/HDDS-2151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Doroszlai, Attila updated HDDS-2151: Status: Patch Available (was: Open) > Ozone client prints the entire request payload in DEBUG level. > -- > > Key: HDDS-2151 > URL: https://issues.apache.org/jira/browse/HDDS-2151 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Aravindan Vijayan >Assignee: Doroszlai, Attila >Priority: Major > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > In XceiverClientRatis.java:221, we have the following snippet where we have a > DEBUG line that prints out the entire Container Request proto. > {code} > ContainerCommandRequestProto finalPayload = > ContainerCommandRequestProto.newBuilder(request) > .setTraceID(TracingUtil.exportCurrentSpan()) > .build(); > boolean isReadOnlyRequest = HddsUtils.isReadOnly(finalPayload); > ByteString byteString = finalPayload.toByteString(); > LOG.debug("sendCommandAsync {} {}", isReadOnlyRequest, finalPayload); > return isReadOnlyRequest ? > getClient().sendReadOnlyAsync(() -> byteString) : > getClient().sendAsync(() -> byteString); > {code} > This causes OOM while writing large (~300MB) keys. > {code} > SLF4J: Failed toString() invocation on an object of type > [org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos$ContainerCommandRequestProto] > Reported exception: > java.lang.OutOfMemoryError: Java heap space > at java.util.Arrays.copyOf(Arrays.java:3332) > at > java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124) > at > java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:649) > at java.lang.StringBuilder.append(StringBuilder.java:202) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormatEscaper.escapeBytes(TextFormatEscaper.java:75) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormatEscaper.escapeBytes(TextFormatEscaper.java:94) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat.escapeBytes(TextFormat.java:1836) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.printFieldValue(TextFormat.java:436) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.printSingleField(TextFormat.java:376) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.printField(TextFormat.java:338) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.print(TextFormat.java:325) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.printFieldValue(TextFormat.java:449) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.printSingleField(TextFormat.java:376) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.printField(TextFormat.java:338) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.print(TextFormat.java:325) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.access$000(TextFormat.java:307) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat.print(TextFormat.java:68) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat.printToString(TextFormat.java:148) > at > org.apache.ratis.thirdparty.com.google.protobuf.AbstractMessage.toString(AbstractMessage.java:117) > at > org.slf4j.helpers.MessageFormatter.safeObjectAppend(MessageFormatter.java:299) > at > org.slf4j.helpers.MessageFormatter.deeplyAppendParameter(MessageFormatter.java:271) > at > org.slf4j.helpers.MessageFormatter.arrayFormat(MessageFormatter.java:233) > at > org.slf4j.helpers.MessageFormatter.arrayFormat(MessageFormatter.java:173) > at org.slf4j.helpers.MessageFormatter.format(MessageFormatter.java:151) > at org.slf4j.impl.Log4jLoggerAdapter.debug(Log4jLoggerAdapter.java:252) > at > org.apache.hadoop.hdds.scm.XceiverClientRatis.sendRequestAsync(XceiverClientRatis.java:221) > at > org.apache.hadoop.hdds.scm.XceiverClientRatis.sendCommandAsync(XceiverClientRatis.java:302) > at > org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.writeChunkAsync(ContainerProtocolCalls.java:310) > at > org.apache.hadoop.hdds.scm.storage.BlockOutputStream.writeChunkToContainer(BlockOutputStream.java:601) > at > org.apache.hadoop.hdds.scm.storage.BlockOutputStream.writeChunk(BlockOutputStream.java:459) > at > org.apache.hadoop.hdds.scm.storage.BlockOutputStream.write(BlockOutputStream.java:240) > at >
[jira] [Updated] (HDDS-2151) Ozone client prints the entire request payload in DEBUG level.
[ https://issues.apache.org/jira/browse/HDDS-2151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-2151: - Labels: pull-request-available (was: ) > Ozone client prints the entire request payload in DEBUG level. > -- > > Key: HDDS-2151 > URL: https://issues.apache.org/jira/browse/HDDS-2151 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Aravindan Vijayan >Assignee: YiSheng Lien >Priority: Major > Labels: pull-request-available > > In XceiverClientRatis.java:221, we have the following snippet where we have a > DEBUG line that prints out the entire Container Request proto. > {code} > ContainerCommandRequestProto finalPayload = > ContainerCommandRequestProto.newBuilder(request) > .setTraceID(TracingUtil.exportCurrentSpan()) > .build(); > boolean isReadOnlyRequest = HddsUtils.isReadOnly(finalPayload); > ByteString byteString = finalPayload.toByteString(); > LOG.debug("sendCommandAsync {} {}", isReadOnlyRequest, finalPayload); > return isReadOnlyRequest ? > getClient().sendReadOnlyAsync(() -> byteString) : > getClient().sendAsync(() -> byteString); > {code} > This causes OOM while writing large (~300MB) keys. > {code} > SLF4J: Failed toString() invocation on an object of type > [org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos$ContainerCommandRequestProto] > Reported exception: > java.lang.OutOfMemoryError: Java heap space > at java.util.Arrays.copyOf(Arrays.java:3332) > at > java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124) > at > java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:649) > at java.lang.StringBuilder.append(StringBuilder.java:202) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormatEscaper.escapeBytes(TextFormatEscaper.java:75) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormatEscaper.escapeBytes(TextFormatEscaper.java:94) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat.escapeBytes(TextFormat.java:1836) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.printFieldValue(TextFormat.java:436) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.printSingleField(TextFormat.java:376) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.printField(TextFormat.java:338) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.print(TextFormat.java:325) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.printFieldValue(TextFormat.java:449) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.printSingleField(TextFormat.java:376) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.printField(TextFormat.java:338) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.print(TextFormat.java:325) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.access$000(TextFormat.java:307) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat.print(TextFormat.java:68) > at > org.apache.ratis.thirdparty.com.google.protobuf.TextFormat.printToString(TextFormat.java:148) > at > org.apache.ratis.thirdparty.com.google.protobuf.AbstractMessage.toString(AbstractMessage.java:117) > at > org.slf4j.helpers.MessageFormatter.safeObjectAppend(MessageFormatter.java:299) > at > org.slf4j.helpers.MessageFormatter.deeplyAppendParameter(MessageFormatter.java:271) > at > org.slf4j.helpers.MessageFormatter.arrayFormat(MessageFormatter.java:233) > at > org.slf4j.helpers.MessageFormatter.arrayFormat(MessageFormatter.java:173) > at org.slf4j.helpers.MessageFormatter.format(MessageFormatter.java:151) > at org.slf4j.impl.Log4jLoggerAdapter.debug(Log4jLoggerAdapter.java:252) > at > org.apache.hadoop.hdds.scm.XceiverClientRatis.sendRequestAsync(XceiverClientRatis.java:221) > at > org.apache.hadoop.hdds.scm.XceiverClientRatis.sendCommandAsync(XceiverClientRatis.java:302) > at > org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.writeChunkAsync(ContainerProtocolCalls.java:310) > at > org.apache.hadoop.hdds.scm.storage.BlockOutputStream.writeChunkToContainer(BlockOutputStream.java:601) > at > org.apache.hadoop.hdds.scm.storage.BlockOutputStream.writeChunk(BlockOutputStream.java:459) > at > org.apache.hadoop.hdds.scm.storage.BlockOutputStream.write(BlockOutputStream.java:240) > at >