[
https://issues.apache.org/jira/browse/PIG-5371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Bodor updated PIG-5371:
------------------------------
Description:
Attached [^simpleTest.out]. It seems like HDFS counter 'HDFS_BYTES_WRITTEN'
returns the byte count not only for the result of pig store operator, but it
includes the jar files as well.
{code}
2018-11-23 10:14:52,661 [IPC Server handler 5 on 54929] INFO
org.apache.hadoop.hdfs.StateChange - BLOCK* allocate blk_1073741827_1003,
replicas=127.0.0.1:54934, 127.0.0.1:54930, 127.0.0.1:54943 for
/tmp/temp-157262781/tmp-1057655772/automaton-1.11-8.jar
...
2018-11-23 10:14:52,735 [PacketResponder:
BP-26001448-10.200.50.195-1542964474138:blk_1073741827_1003,
type=HAS_DOWNSTREAM_IN_PIPELINE, downstreams=2:[127.0.0.1:54930,
127.0.0.1:54943]] INFO
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace - src:
/127.0.0.1:54978, dest: /127.0.0.1:54934, bytes: 176285, op: HDFS_WRITE, cliID:
DFSClient_NONMAPREDUCE_-1959727442_1, offset: 0, srvID:
108c4000-1ae0-402e-82cf-bf403629c0f7, blockid:
BP-26001448-10.200.50.195-1542964474138:blk_1073741827_1003, duration(ns):
57162859
{code}
was:
Attached [^simpleTest.out]. It seems like HDFS counter 'HDFS_BYTES_WRITTEN'
returns the byte count not only for the result of pig store operator, but it
includes the jar files as well.
{code}
{code}
> Hdfs bytes written assertions fail in TestPigRunner
> ---------------------------------------------------
>
> Key: PIG-5371
> URL: https://issues.apache.org/jira/browse/PIG-5371
> Project: Pig
> Issue Type: Bug
> Reporter: Laszlo Bodor
> Assignee: Laszlo Bodor
> Priority: Major
> Attachments: PIG-5371.01.patch, simpleTest.out
>
>
> Attached [^simpleTest.out]. It seems like HDFS counter 'HDFS_BYTES_WRITTEN'
> returns the byte count not only for the result of pig store operator, but it
> includes the jar files as well.
> {code}
> 2018-11-23 10:14:52,661 [IPC Server handler 5 on 54929] INFO
> org.apache.hadoop.hdfs.StateChange - BLOCK* allocate blk_1073741827_1003,
> replicas=127.0.0.1:54934, 127.0.0.1:54930, 127.0.0.1:54943 for
> /tmp/temp-157262781/tmp-1057655772/automaton-1.11-8.jar
> ...
> 2018-11-23 10:14:52,735 [PacketResponder:
> BP-26001448-10.200.50.195-1542964474138:blk_1073741827_1003,
> type=HAS_DOWNSTREAM_IN_PIPELINE, downstreams=2:[127.0.0.1:54930,
> 127.0.0.1:54943]] INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace - src:
> /127.0.0.1:54978, dest: /127.0.0.1:54934, bytes: 176285, op: HDFS_WRITE,
> cliID: DFSClient_NONMAPREDUCE_-1959727442_1, offset: 0, srvID:
> 108c4000-1ae0-402e-82cf-bf403629c0f7, blockid:
> BP-26001448-10.200.50.195-1542964474138:blk_1073741827_1003, duration(ns):
> 57162859
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
