[
https://issues.apache.org/jira/browse/HADOOP-8648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13428514#comment-13428514
]
Todd Lipcon commented on HADOOP-8648:
-------------------------------------
{code}
+ if (unlikely((uintptr_t)(sums_addr + sums_offset) & 0x3)) {
+ char buf[256];
+ snprintf(buf, sizeof(buf), "sums_addr + sums_offset must be aligned "
+ "to a 4-byte boundary! sums_addr = %p, sums_offset = %d.",
+ sums_addr, sums_offset);
+ THROW(env, "java/lang/IllegalArgumentException", buf);
+ return;
+ }
{code}
Is that true? I don't think we always fit this requirement. I thought the
native code worked even with unaligned pointers.
----
- Can you add a direct unit test to TestDataChecksum for these cases? I don't
think testing this from HDFS is necessarily the best way when you could trigger
the condition explicitly in the checksum code.
I'm also a little confused by the changes in the C code. eg:
{code}
- * block_size : The size of each block in bytes.
+ * block_size : The size of each block in bytes. Must be >= 8
{code}
Why is that the case? It seems like the code path should still work fine there,
just setting counter = 0 and skipping the initial while loop.
> libhadoop: native CRC32 validation crashes when io.bytes.per.checksum=1
> ------------------------------------------------------------------------
>
> Key: HADOOP-8648
> URL: https://issues.apache.org/jira/browse/HADOOP-8648
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 2.0.0-alpha
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Attachments: HADOOP-8648.001.patch, HADOOP-8648.002.patch
>
>
> The native CRC32 code, found in {{pipelined_crc32c}}, crashes when blocksize
> is set to 1.
> {code}
> 12:27:14,886 INFO NativeCodeLoader:50 - Loaded the native-hadoop library
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> # SIGSEGV (0xb) at pc=0x00007fa00ee5a340, pid=24100, tid=140326058854144
> #
> # JRE version: 6.0_29-b11
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (20.4-b02 mixed mode linux-amd64
> compressed oops)
> # Problematic frame:
> # C [libhadoop.so.1.0.0+0x8340] pipelined_crc32c+0xa0
> #
> # An error report file with more information is saved as:
> # /h/hs_err_pid24100.log
> #
> # If you would like to submit a bug report, please visit:
> # http://java.sun.com/webapps/bugreport/crash.jsp
> #
> Aborted
> {code}
> The Java CRC code works fine in this case.
> Choosing blocksize=1 is a __very__ odd choice. It means that we're storing a
> 4-byte checksum for every byte.
> {code}
> -rw-r--r-- 1 cmccabe users 49398 Aug 3 11:33 blk_4702510289566780538
> -rw-r--r-- 1 cmccabe users 197599 Aug 3 11:33
> blk_4702510289566780538_1199.meta
> {code}
> However, obviously crashing is never the right thing to do.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
