[
https://issues.apache.org/jira/browse/HADOOP-15822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16642104#comment-16642104
]
Jason Lowe commented on HADOOP-15822:
-------------------------------------
bq. do you think it's related? Or is it something different, maybe MR-specific?
I do not think it is related. The MapOutput buffer code is miscalculating how
much buffer space is remaining before it forces a spill. In this failure case
the buffer involved is not dealing with compressed data, so it should not
matter what codec is being used. Have you tried reproducing it with lz4 or no
codec at all?
I'll dig a bit into the Jenkins test failures to see if they are somehow
related.
> zstd compressor can fail with a small output buffer
> ---------------------------------------------------
>
> Key: HADOOP-15822
> URL: https://issues.apache.org/jira/browse/HADOOP-15822
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 2.9.0, 3.0.0
> Reporter: Jason Lowe
> Assignee: Jason Lowe
> Priority: Major
> Attachments: HADOOP-15822.001.patch, HADOOP-15822.002.patch
>
>
> TestZStandardCompressorDecompressor fails a couple of tests on my machine
> with the latest zstd library (1.3.5). Compression can fail to successfully
> finalize the stream when a small output buffer is used resulting in a failed
> to init error, and decompression with a direct buffer can fail with an
> invalid src size error.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
