[jira] [Commented] (HDDS-1120) Add a config to disable checksum verification during read even though checksum data is present in the persisted data
[ https://issues.apache.org/jira/browse/HDDS-1120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16776133#comment-16776133 ] Hudson commented on HDDS-1120: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #16044 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/16044/]) HDDS-1120. Add a config to disable checksum verification during read. (github: rev 106bdc6c0474480654aa27cd7b8b50342f69e707) * (edit) hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestOzoneRpcClientAbstract.java * (edit) hadoop-hdds/common/src/main/resources/ozone-default.xml * (edit) hadoop-ozone/objectstore-service/src/main/java/org/apache/hadoop/ozone/web/storage/DistributedStorageHandler.java * (edit) hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/io/KeyInputStream.java * (edit) hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConfigKeys.java * (edit) hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/rpc/RpcClient.java * (edit) hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BlockInputStream.java * (edit) hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/TestChunkStreams.java > Add a config to disable checksum verification during read even though > checksum data is present in the persisted data > > > Key: HDDS-1120 > URL: https://issues.apache.org/jira/browse/HDDS-1120 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Client >Affects Versions: 0.4.0 >Reporter: Shashikant Banerjee >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Fix For: 0.4.0 > > Attachments: HDDS-1120.00.patch > > Time Spent: 40m > Remaining Estimate: 0h > > Currently, if the checksum is computed during data write and persisted in the > disk, we will always end up verifying it while reading. This Jira aims to > selectively disable checksum verification during reads even though checksum > info is present in the data stored. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1120) Add a config to disable checksum verification during read even though checksum data is present in the persisted data
[ https://issues.apache.org/jira/browse/HDDS-1120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16776130#comment-16776130 ] Bharat Viswanadham commented on HDDS-1120: -- Thank You [~linyiqun] for the review. I have committed this to trunk. For ease of review I have done as 2 commits. During commit to trunk, I did squash and merge, it will be showing as a single commit. > Add a config to disable checksum verification during read even though > checksum data is present in the persisted data > > > Key: HDDS-1120 > URL: https://issues.apache.org/jira/browse/HDDS-1120 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Client >Affects Versions: 0.4.0 >Reporter: Shashikant Banerjee >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Fix For: 0.4.0 > > Attachments: HDDS-1120.00.patch > > Time Spent: 40m > Remaining Estimate: 0h > > Currently, if the checksum is computed during data write and persisted in the > disk, we will always end up verifying it while reading. This Jira aims to > selectively disable checksum verification during reads even though checksum > info is present in the data stored. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1120) Add a config to disable checksum verification during read even though checksum data is present in the persisted data
[ https://issues.apache.org/jira/browse/HDDS-1120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775879#comment-16775879 ] Yiqun Lin commented on HDDS-1120: - LGTM, +1. By the PR way, I see there are two commits in the PR. Is this will still show two commit when merged to trunk? Need we combine these two commit to a final commit with the corresponding message? > Add a config to disable checksum verification during read even though > checksum data is present in the persisted data > > > Key: HDDS-1120 > URL: https://issues.apache.org/jira/browse/HDDS-1120 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Client >Affects Versions: 0.4.0 >Reporter: Shashikant Banerjee >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Fix For: 0.4.0 > > Attachments: HDDS-1120.00.patch > > Time Spent: 0.5h > Remaining Estimate: 0h > > Currently, if the checksum is computed during data write and persisted in the > disk, we will always end up verifying it while reading. This Jira aims to > selectively disable checksum verification during reads even though checksum > info is present in the data stored. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1120) Add a config to disable checksum verification during read even though checksum data is present in the persisted data
[ https://issues.apache.org/jira/browse/HDDS-1120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775627#comment-16775627 ] Bharat Viswanadham commented on HDDS-1120: -- Thank You [~linyiqun] for the review. Addressed your review comment and updated the PR. > Add a config to disable checksum verification during read even though > checksum data is present in the persisted data > > > Key: HDDS-1120 > URL: https://issues.apache.org/jira/browse/HDDS-1120 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Client >Affects Versions: 0.4.0 >Reporter: Shashikant Banerjee >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Fix For: 0.4.0 > > Attachments: HDDS-1120.00.patch > > Time Spent: 0.5h > Remaining Estimate: 0h > > Currently, if the checksum is computed during data write and persisted in the > disk, we will always end up verifying it while reading. This Jira aims to > selectively disable checksum verification during reads even though checksum > info is present in the data stored. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1120) Add a config to disable checksum verification during read even though checksum data is present in the persisted data
[
https://issues.apache.org/jira/browse/HDDS-1120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775055#comment-16775055
]
Yiqun Lin commented on HDDS-1120:
-
Thanks for addressing the comment, [~bharatviswa]! Only one small suggestion:
Can we reuse common lines in two test case? I mean we can extract follow lines
to a method.
{code:java}
String volumeName = UUID.randomUUID().toString();
String bucketName = UUID.randomUUID().toString();
String value = "sample value";
store.createVolume(volumeName);
OzoneVolume volume = store.getVolume(volumeName);
volume.createBucket(bucketName);
OzoneBucket bucket = volume.getBucket(bucketName);
String keyName = UUID.randomUUID().toString();
// Write data into a key
OzoneOutputStream out = bucket.createKey(keyName,
value.getBytes().length, ReplicationType.RATIS,
ReplicationFactor.ONE, new HashMap<>());
out.write(value.getBytes());
out.close();
// We need to find the location of the chunk file corresponding to the
// data we just wrote.
OzoneKey key = bucket.getKey(keyName);
long containerID = ((OzoneKeyDetails) key).getOzoneKeyLocations().get(0)
.getContainerID();
// Get the container by traversing the datanodes. Atleast one of the
// datanode must have this container.
Container container = null;
...
Assert.assertNotNull("Container not found", container);
{code}
> Add a config to disable checksum verification during read even though
> checksum data is present in the persisted data
>
>
> Key: HDDS-1120
> URL: https://issues.apache.org/jira/browse/HDDS-1120
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: Ozone Client
>Affects Versions: 0.4.0
>Reporter: Shashikant Banerjee
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.4.0
>
> Attachments: HDDS-1120.00.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Currently, if the checksum is computed during data write and persisted in the
> disk, we will always end up verifying it while reading. This Jira aims to
> selectively disable checksum verification during reads even though checksum
> info is present in the data stored.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1120) Add a config to disable checksum verification during read even though checksum data is present in the persisted data
[ https://issues.apache.org/jira/browse/HDDS-1120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16774491#comment-16774491 ] Bharat Viswanadham commented on HDDS-1120: -- Thank You [~linyiqun] for the review. Opened a PR and addressed all the review comments. As Jenkins run is not running for some patches, opened a PR where Jenkins run is set up for PR's. > Add a config to disable checksum verification during read even though > checksum data is present in the persisted data > > > Key: HDDS-1120 > URL: https://issues.apache.org/jira/browse/HDDS-1120 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Client >Affects Versions: 0.4.0 >Reporter: Shashikant Banerjee >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Fix For: 0.4.0 > > Attachments: HDDS-1120.00.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Currently, if the checksum is computed during data write and persisted in the > disk, we will always end up verifying it while reading. This Jira aims to > selectively disable checksum verification during reads even though checksum > info is present in the data stored. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1120) Add a config to disable checksum verification during read even though checksum data is present in the persisted data
[ https://issues.apache.org/jira/browse/HDDS-1120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16770393#comment-16770393 ] Yiqun Lin commented on HDDS-1120: - The patch almost looks good to me, [~bharatviswa]. Would you mind adding an unit test for the case of JIRA description mentioned? Will like following behaviour: With checksum persisted, then corrupt the data * If enabled the checksum verification (by default way), the read operation will fail. * If disabled, the read operation should be successful. > Add a config to disable checksum verification during read even though > checksum data is present in the persisted data > > > Key: HDDS-1120 > URL: https://issues.apache.org/jira/browse/HDDS-1120 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Client >Affects Versions: 0.4.0 >Reporter: Shashikant Banerjee >Assignee: Bharat Viswanadham >Priority: Major > Fix For: 0.4.0 > > Attachments: HDDS-1120.00.patch > > > Currently, if the checksum is computed during data write and persisted in the > disk, we will always end up verifying it while reading. This Jira aims to > selectively disable checksum verification during reads even though checksum > info is present in the data stored. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1120) Add a config to disable checksum verification during read even though checksum data is present in the persisted data
[
https://issues.apache.org/jira/browse/HDDS-1120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16770169#comment-16770169
]
Hadoop QA commented on HDDS-1120:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 7s{color}
| {color:red} HDDS-1120 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDDS-1120 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12958982/HDDS-1120.00.patch |
| Console output |
https://builds.apache.org/job/PreCommit-HDDS-Build/2296/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Add a config to disable checksum verification during read even though
> checksum data is present in the persisted data
>
>
> Key: HDDS-1120
> URL: https://issues.apache.org/jira/browse/HDDS-1120
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: Ozone Client
>Affects Versions: 0.4.0
>Reporter: Shashikant Banerjee
>Assignee: Bharat Viswanadham
>Priority: Major
> Fix For: 0.4.0
>
> Attachments: HDDS-1120.00.patch
>
>
> Currently, if the checksum is computed during data write and persisted in the
> disk, we will always end up verifying it while reading. This Jira aims to
> selectively disable checksum verification during reads even though checksum
> info is present in the data stored.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
