[jira] [Updated] (HDFS-5634) allow BlockReaderLocal to switch between checksumming and not
[ https://issues.apache.org/jira/browse/HDFS-5634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-5634: --- Status: Open (was: Patch Available) allow BlockReaderLocal to switch between checksumming and not - Key: HDFS-5634 URL: https://issues.apache.org/jira/browse/HDFS-5634 Project: Hadoop HDFS Issue Type: Sub-task Components: hdfs-client Affects Versions: 3.0.0 Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Attachments: HDFS-5634.001.patch, HDFS-5634.002.patch, HDFS-5634.003.patch, HDFS-5634.004.patch, HDFS-5634.005.patch, HDFS-5634.006.patch, HDFS-5634.007.patch, HDFS-5634.008.patch BlockReaderLocal should be able to switch between checksumming and non-checksumming, so that when we get notifications that something is mlocked (see HDFS-5182), we can avoid checksumming when reading from that block. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Updated] (HDFS-5634) allow BlockReaderLocal to switch between checksumming and not
[ https://issues.apache.org/jira/browse/HDFS-5634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-5634: --- Attachment: HDFS-5634.009.branch-2.patch posted patch for branch-2 allow BlockReaderLocal to switch between checksumming and not - Key: HDFS-5634 URL: https://issues.apache.org/jira/browse/HDFS-5634 Project: Hadoop HDFS Issue Type: Sub-task Components: hdfs-client Affects Versions: 3.0.0 Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Attachments: HDFS-5634.001.patch, HDFS-5634.002.patch, HDFS-5634.003.patch, HDFS-5634.004.patch, HDFS-5634.005.patch, HDFS-5634.006.patch, HDFS-5634.007.patch, HDFS-5634.008.patch, HDFS-5634.009.branch-2.patch BlockReaderLocal should be able to switch between checksumming and non-checksumming, so that when we get notifications that something is mlocked (see HDFS-5182), we can avoid checksumming when reading from that block. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Updated] (HDFS-5634) allow BlockReaderLocal to switch between checksumming and not
[ https://issues.apache.org/jira/browse/HDFS-5634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-5634: --- Target Version/s: 2.4.0 (was: 3.0.0) Affects Version/s: (was: 3.0.0) 2.4.0 Fix Version/s: 2.4.0 backported to 2.4 allow BlockReaderLocal to switch between checksumming and not - Key: HDFS-5634 URL: https://issues.apache.org/jira/browse/HDFS-5634 Project: Hadoop HDFS Issue Type: Sub-task Components: hdfs-client Affects Versions: 2.4.0 Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Fix For: 2.4.0 Attachments: HDFS-5634.001.patch, HDFS-5634.002.patch, HDFS-5634.003.patch, HDFS-5634.004.patch, HDFS-5634.005.patch, HDFS-5634.006.patch, HDFS-5634.007.patch, HDFS-5634.008.patch, HDFS-5634.009.branch-2.patch BlockReaderLocal should be able to switch between checksumming and non-checksumming, so that when we get notifications that something is mlocked (see HDFS-5182), we can avoid checksumming when reading from that block. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Updated] (HDFS-5634) allow BlockReaderLocal to switch between checksumming and not
[ https://issues.apache.org/jira/browse/HDFS-5634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-5634: --- Attachment: HDFS-5634.008.patch allow BlockReaderLocal to switch between checksumming and not - Key: HDFS-5634 URL: https://issues.apache.org/jira/browse/HDFS-5634 Project: Hadoop HDFS Issue Type: Sub-task Components: hdfs-client Affects Versions: 3.0.0 Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Attachments: HDFS-5634.001.patch, HDFS-5634.002.patch, HDFS-5634.003.patch, HDFS-5634.004.patch, HDFS-5634.005.patch, HDFS-5634.006.patch, HDFS-5634.007.patch, HDFS-5634.008.patch BlockReaderLocal should be able to switch between checksumming and non-checksumming, so that when we get notifications that something is mlocked (see HDFS-5182), we can avoid checksumming when reading from that block. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Updated] (HDFS-5634) allow BlockReaderLocal to switch between checksumming and not
[ https://issues.apache.org/jira/browse/HDFS-5634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-5634: --- Attachment: HDFS-5634.006.patch * Fixed a bug introduced by converting to using skipChecksums in some places. * Ensure that the fast path starts reading on a checksum boundary. allow BlockReaderLocal to switch between checksumming and not - Key: HDFS-5634 URL: https://issues.apache.org/jira/browse/HDFS-5634 Project: Hadoop HDFS Issue Type: Sub-task Components: hdfs-client Affects Versions: 3.0.0 Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Attachments: HDFS-5634.001.patch, HDFS-5634.002.patch, HDFS-5634.003.patch, HDFS-5634.004.patch, HDFS-5634.005.patch, HDFS-5634.006.patch BlockReaderLocal should be able to switch between checksumming and non-checksumming, so that when we get notifications that something is mlocked (see HDFS-5182), we can avoid checksumming when reading from that block. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Updated] (HDFS-5634) allow BlockReaderLocal to switch between checksumming and not
[ https://issues.apache.org/jira/browse/HDFS-5634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-5634: --- Attachment: HDFS-5634.007.patch allow BlockReaderLocal to switch between checksumming and not - Key: HDFS-5634 URL: https://issues.apache.org/jira/browse/HDFS-5634 Project: Hadoop HDFS Issue Type: Sub-task Components: hdfs-client Affects Versions: 3.0.0 Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Attachments: HDFS-5634.001.patch, HDFS-5634.002.patch, HDFS-5634.003.patch, HDFS-5634.004.patch, HDFS-5634.005.patch, HDFS-5634.006.patch, HDFS-5634.007.patch BlockReaderLocal should be able to switch between checksumming and non-checksumming, so that when we get notifications that something is mlocked (see HDFS-5182), we can avoid checksumming when reading from that block. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Updated] (HDFS-5634) allow BlockReaderLocal to switch between checksumming and not
[ https://issues.apache.org/jira/browse/HDFS-5634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-5634: --- Attachment: HDFS-5634.005.patch allow BlockReaderLocal to switch between checksumming and not - Key: HDFS-5634 URL: https://issues.apache.org/jira/browse/HDFS-5634 Project: Hadoop HDFS Issue Type: Sub-task Components: hdfs-client Affects Versions: 3.0.0 Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Attachments: HDFS-5634.001.patch, HDFS-5634.002.patch, HDFS-5634.003.patch, HDFS-5634.004.patch, HDFS-5634.005.patch BlockReaderLocal should be able to switch between checksumming and non-checksumming, so that when we get notifications that something is mlocked (see HDFS-5182), we can avoid checksumming when reading from that block. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Updated] (HDFS-5634) allow BlockReaderLocal to switch between checksumming and not
[ https://issues.apache.org/jira/browse/HDFS-5634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-5634: --- Attachment: HDFS-5634.003.patch allow BlockReaderLocal to switch between checksumming and not - Key: HDFS-5634 URL: https://issues.apache.org/jira/browse/HDFS-5634 Project: Hadoop HDFS Issue Type: Sub-task Components: hdfs-client Affects Versions: 3.0.0 Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Attachments: HDFS-5634.001.patch, HDFS-5634.002.patch, HDFS-5634.003.patch BlockReaderLocal should be able to switch between checksumming and non-checksumming, so that when we get notifications that something is mlocked (see HDFS-5182), we can avoid checksumming when reading from that block. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Updated] (HDFS-5634) allow BlockReaderLocal to switch between checksumming and not
[ https://issues.apache.org/jira/browse/HDFS-5634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-5634: --- Attachment: HDFS-5634.004.patch I optimized the CPU consumption a bit by caching the checksum size and bytes-per in final ints, and avoiding the need to re-do some multiplications a few times on every read. perf stat now gives me 305,384,306,460 cycles for TestParallelShortCircuitRead, as opposed to 321,040,227,686 cycles before. allow BlockReaderLocal to switch between checksumming and not - Key: HDFS-5634 URL: https://issues.apache.org/jira/browse/HDFS-5634 Project: Hadoop HDFS Issue Type: Sub-task Components: hdfs-client Affects Versions: 3.0.0 Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Attachments: HDFS-5634.001.patch, HDFS-5634.002.patch, HDFS-5634.003.patch, HDFS-5634.004.patch BlockReaderLocal should be able to switch between checksumming and non-checksumming, so that when we get notifications that something is mlocked (see HDFS-5182), we can avoid checksumming when reading from that block. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Updated] (HDFS-5634) allow BlockReaderLocal to switch between checksumming and not
[
https://issues.apache.org/jira/browse/HDFS-5634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin Patrick McCabe updated HDFS-5634:
---
Attachment: HDFS-5634.001.patch
This change allows the BlockReaderLocal to switch back and forth between
mlocked and non-mlocked states. (Later, we will hook up callbacks from code
added by HDFS-5182 to do this).
A few other changes:
* honor the readahead parameter, so that skipping checksums doesn't mean
skipping all buffering. See HDFS-4710 of the discussion of why we want this.
* for reads to a direct ByteBuffer, add a fast lane that copies directly into
the user-supplied ByteBuffer. We only do this if the read is longer than our
configured readahead. This avoids a copy.
* use pread everywhere instead of read. This means that if a client opens a
file multiple times, they only need one set of file descriptors rather than
multiple. This will become more important with HDFS-5182, since that change
will add a notification system per set of FDs. We don't want to track too many
of those.
* move reading of the meta file header out of the {{BlockReaderLocal}}
constructor. This will allow us to implement HDFS-4960 (only read version
once). This is mainly a win in the no-checksum case.
* avoid using a skip buffer in BlockReaderLocal#skip (implements HDFS-5574)
allow BlockReaderLocal to switch between checksumming and not
-
Key: HDFS-5634
URL: https://issues.apache.org/jira/browse/HDFS-5634
Project: Hadoop HDFS
Issue Type: Sub-task
Components: hdfs-client
Affects Versions: 3.0.0
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Attachments: HDFS-5634.001.patch
BlockReaderLocal should be able to switch between checksumming and
non-checksumming, so that when we get notifications that something is mlocked
(see HDFS-5182), we can avoid checksumming when reading from that block.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HDFS-5634) allow BlockReaderLocal to switch between checksumming and not
[ https://issues.apache.org/jira/browse/HDFS-5634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-5634: --- Status: Patch Available (was: Open) allow BlockReaderLocal to switch between checksumming and not - Key: HDFS-5634 URL: https://issues.apache.org/jira/browse/HDFS-5634 Project: Hadoop HDFS Issue Type: Sub-task Components: hdfs-client Affects Versions: 3.0.0 Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Attachments: HDFS-5634.001.patch BlockReaderLocal should be able to switch between checksumming and non-checksumming, so that when we get notifications that something is mlocked (see HDFS-5182), we can avoid checksumming when reading from that block. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5634) allow BlockReaderLocal to switch between checksumming and not
[
https://issues.apache.org/jira/browse/HDFS-5634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin Patrick McCabe updated HDFS-5634:
---
Attachment: HDFS-5634.002.patch
* fix findbugs warnings
* fix bug in {{BlockReaderLocal#skip}}
* fix bug causing mmaps to be granted even without checksum-skipping in effect
allow BlockReaderLocal to switch between checksumming and not
-
Key: HDFS-5634
URL: https://issues.apache.org/jira/browse/HDFS-5634
Project: Hadoop HDFS
Issue Type: Sub-task
Components: hdfs-client
Affects Versions: 3.0.0
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Attachments: HDFS-5634.001.patch, HDFS-5634.002.patch
BlockReaderLocal should be able to switch between checksumming and
non-checksumming, so that when we get notifications that something is mlocked
(see HDFS-5182), we can avoid checksumming when reading from that block.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
