If I recall this problem correctly, the root cause is the default zstd
compression block size is 256kb, and Hadoop Zstd compression will attempt
to use the OS platform default compression size, if it is available. The
recommended output size is slightly bigger than input size to account for
header
Thanks for the pointer, it does look similar. However we are roughly on the
latest of branch-3.1 and this fix is in our branch. I'm pretty sure we have
all the zstd fixes.
I believe the libzstd version used is 1.4.4 but need to confirm. I
suspected it's a library version issue because we've been u
Hi Wei Chiu,
What is the Hadoop version being used?
Give a check if HADOOP-15822 is there, it had something similar error.
-Ayush
> On 11-May-2020, at 10:11 PM, Wei-Chiu Chuang wrote:
>
> Hadoop devs,
>
> A colleague of mine recently hit a strange issue where zstd compression
> codec crashes.
For more details, see
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/137/
[May 10, 2020 6:13:30 AM] (Ayush Saxena) HDFS-15250. Setting
`dfs.client.use.datanode.hostname` to true can crash the system because of
unhandled UnresolvedAddressException. Contributed by Ctest.
Hadoop devs,
A colleague of mine recently hit a strange issue where zstd compression
codec crashes.
Caused by: java.lang.InternalError: Error (generic)
at
org.apache.hadoop.io.compress.zstd.ZStandardCompressor.deflateBytesDirect(Native
Method)
at
org.apache.hadoop.io.compress.zstd.ZStandardCompre
[
https://issues.apache.org/jira/browse/HADOOP-17033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang resolved HADOOP-17033.
--
Fix Version/s: 3.4.0
Resolution: Fixed
> Update commons-codec from 1.11 to 1.1
For more details, see
https://builds.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86/682/
No changes
-1 overall
The following subsystems voted -1:
asflicense findbugs hadolint pathlen unit xml
The following subsystems voted -1 but
were configured to be filtered/ignored:
cc c
Steve Loughran created HADOOP-17036:
---
Summary: TestFTPFileSystem failing as ftp server dir already exists
Key: HADOOP-17036
URL: https://issues.apache.org/jira/browse/HADOOP-17036
Project: Hadoop Com
[
https://issues.apache.org/jira/browse/HADOOP-17034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Masatake Iwasaki resolved HADOOP-17034.
---
Assignee: (was: Masatake Iwasaki)
Resolution: Duplicate
closing this as d
