[jira] [Updated] (HDFS-6147) New blocks scanning will be delayed due to issue in BlockPoolSliceScanner#updateBytesToScan(..)
[
https://issues.apache.org/jira/browse/HDFS-6147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinayakumar B updated HDFS-6147:
Resolution: Won't Fix
Status: Resolved (was: Patch Available)
Since the block scanner is re-written, this issue no longer holds.
> New blocks scanning will be delayed due to issue in
> BlockPoolSliceScanner#updateBytesToScan(..)
> ---
>
> Key: HDFS-6147
> URL: https://issues.apache.org/jira/browse/HDFS-6147
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.4.0
>Reporter: Vinayakumar B
>Assignee: Vinayakumar B
> Attachments: HDFS-6147.patch, HDFS-6147.patch, HDFS-6147.patch,
> HDFS-6147.patch, HDFS-6147.patch, HDFS-6147.patch
>
>
> New blocks scanning will be delayed if old blocks deleted after datanode
> restart.
> Steps:
> 1. Write some blocks and wait till all scans over
> 2. Restart the datanode
> 3. Delete some of the blocks
> 4. Write new blocks which are less in size compared to deleted blocks.
> Problem:
> {{BlockPoolSliceScanner#updateBytesToScan(..)}} updates {{bytesLeft}} based
> on following comparison
> {code} if (lastScanTime < currentPeriodStart) {
> bytesLeft += len;
> }{code}
> But in {{BlockPoolSliceScanner#assignInitialVerificationTimes()}}
> {{bytesLeft}} decremented using below comparison
> {code}if (now - entry.verificationTime < scanPeriod) {{code}
> Hence when the old blocks are deleted {{bytesLeft}} going negative.
> new blocks will not be scanned until it becomes positive again.
> So in both places verificationtime should be compared against scanperiod.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-6147) New blocks scanning will be delayed due to issue in BlockPoolSliceScanner#updateBytesToScan(..)
[
https://issues.apache.org/jira/browse/HDFS-6147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinayakumar B updated HDFS-6147:
Attachment: HDFS-6147.patch
Rebased the patch.
Hi [~umamaheswararao], can you take a look at the patch?
> New blocks scanning will be delayed due to issue in
> BlockPoolSliceScanner#updateBytesToScan(..)
> ---
>
> Key: HDFS-6147
> URL: https://issues.apache.org/jira/browse/HDFS-6147
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.4.0
>Reporter: Vinayakumar B
>Assignee: Vinayakumar B
> Attachments: HDFS-6147.patch, HDFS-6147.patch, HDFS-6147.patch,
> HDFS-6147.patch, HDFS-6147.patch, HDFS-6147.patch
>
>
> New blocks scanning will be delayed if old blocks deleted after datanode
> restart.
> Steps:
> 1. Write some blocks and wait till all scans over
> 2. Restart the datanode
> 3. Delete some of the blocks
> 4. Write new blocks which are less in size compared to deleted blocks.
> Problem:
> {{BlockPoolSliceScanner#updateBytesToScan(..)}} updates {{bytesLeft}} based
> on following comparison
> {code} if (lastScanTime < currentPeriodStart) {
> bytesLeft += len;
> }{code}
> But in {{BlockPoolSliceScanner#assignInitialVerificationTimes()}}
> {{bytesLeft}} decremented using below comparison
> {code}if (now - entry.verificationTime < scanPeriod) {{code}
> Hence when the old blocks are deleted {{bytesLeft}} going negative.
> new blocks will not be scanned until it becomes positive again.
> So in both places verificationtime should be compared against scanperiod.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-6147) New blocks scanning will be delayed due to issue in BlockPoolSliceScanner#updateBytesToScan(..)
[
https://issues.apache.org/jira/browse/HDFS-6147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinayakumar B updated HDFS-6147:
Attachment: HDFS-6147.patch
rebased and Updated test.
Please review
> New blocks scanning will be delayed due to issue in
> BlockPoolSliceScanner#updateBytesToScan(..)
> ---
>
> Key: HDFS-6147
> URL: https://issues.apache.org/jira/browse/HDFS-6147
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.4.0
>Reporter: Vinayakumar B
>Assignee: Vinayakumar B
> Attachments: HDFS-6147.patch, HDFS-6147.patch, HDFS-6147.patch,
> HDFS-6147.patch, HDFS-6147.patch
>
>
> New blocks scanning will be delayed if old blocks deleted after datanode
> restart.
> Steps:
> 1. Write some blocks and wait till all scans over
> 2. Restart the datanode
> 3. Delete some of the blocks
> 4. Write new blocks which are less in size compared to deleted blocks.
> Problem:
> {{BlockPoolSliceScanner#updateBytesToScan(..)}} updates {{bytesLeft}} based
> on following comparison
> {code} if (lastScanTime < currentPeriodStart) {
> bytesLeft += len;
> }{code}
> But in {{BlockPoolSliceScanner#assignInitialVerificationTimes()}}
> {{bytesLeft}} decremented using below comparison
> {code}if (now - entry.verificationTime < scanPeriod) {{code}
> Hence when the old blocks are deleted {{bytesLeft}} going negative.
> new blocks will not be scanned until it becomes positive again.
> So in both places verificationtime should be compared against scanperiod.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6147) New blocks scanning will be delayed due to issue in BlockPoolSliceScanner#updateBytesToScan(..)
[
https://issues.apache.org/jira/browse/HDFS-6147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinayakumar B updated HDFS-6147:
Attachment: HDFS-6147.patch
Rebased the patch on latest trunk.
> New blocks scanning will be delayed due to issue in
> BlockPoolSliceScanner#updateBytesToScan(..)
> ---
>
> Key: HDFS-6147
> URL: https://issues.apache.org/jira/browse/HDFS-6147
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.4.0
>Reporter: Vinayakumar B
>Assignee: Vinayakumar B
> Attachments: HDFS-6147.patch, HDFS-6147.patch, HDFS-6147.patch,
> HDFS-6147.patch
>
>
> New blocks scanning will be delayed if old blocks deleted after datanode
> restart.
> Steps:
> 1. Write some blocks and wait till all scans over
> 2. Restart the datanode
> 3. Delete some of the blocks
> 4. Write new blocks which are less in size compared to deleted blocks.
> Problem:
> {{BlockPoolSliceScanner#updateBytesToScan(..)}} updates {{bytesLeft}} based
> on following comparison
> {code} if (lastScanTime < currentPeriodStart) {
> bytesLeft += len;
> }{code}
> But in {{BlockPoolSliceScanner#assignInitialVerificationTimes()}}
> {{bytesLeft}} decremented using below comparison
> {code}if (now - entry.verificationTime < scanPeriod) {{code}
> Hence when the old blocks are deleted {{bytesLeft}} going negative.
> new blocks will not be scanned until it becomes positive again.
> So in both places verificationtime should be compared against scanperiod.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6147) New blocks scanning will be delayed due to issue in BlockPoolSliceScanner#updateBytesToScan(..)
[
https://issues.apache.org/jira/browse/HDFS-6147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinayakumar B updated HDFS-6147:
Attachment: HDFS-6147.patch
Addressed above comments
> New blocks scanning will be delayed due to issue in
> BlockPoolSliceScanner#updateBytesToScan(..)
> ---
>
> Key: HDFS-6147
> URL: https://issues.apache.org/jira/browse/HDFS-6147
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.4.0
>Reporter: Vinayakumar B
>Assignee: Vinayakumar B
> Attachments: HDFS-6147.patch, HDFS-6147.patch, HDFS-6147.patch
>
>
> New blocks scanning will be delayed if old blocks deleted after datanode
> restart.
> Steps:
> 1. Write some blocks and wait till all scans over
> 2. Restart the datanode
> 3. Delete some of the blocks
> 4. Write new blocks which are less in size compared to deleted blocks.
> Problem:
> {{BlockPoolSliceScanner#updateBytesToScan(..)}} updates {{bytesLeft}} based
> on following comparison
> {code} if (lastScanTime < currentPeriodStart) {
> bytesLeft += len;
> }{code}
> But in {{BlockPoolSliceScanner#assignInitialVerificationTimes()}}
> {{bytesLeft}} decremented using below comparison
> {code}if (now - entry.verificationTime < scanPeriod) {{code}
> Hence when the old blocks are deleted {{bytesLeft}} going negative.
> new blocks will not be scanned until it becomes positive again.
> So in both places verificationtime should be compared against scanperiod.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6147) New blocks scanning will be delayed due to issue in BlockPoolSliceScanner#updateBytesToScan(..)
[
https://issues.apache.org/jira/browse/HDFS-6147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinayakumar B updated HDFS-6147:
Attachment: HDFS-6147.patch
Attaching the patch with Test failures fix.
> New blocks scanning will be delayed due to issue in
> BlockPoolSliceScanner#updateBytesToScan(..)
> ---
>
> Key: HDFS-6147
> URL: https://issues.apache.org/jira/browse/HDFS-6147
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.4.0
>Reporter: Vinayakumar B
>Assignee: Vinayakumar B
> Attachments: HDFS-6147.patch, HDFS-6147.patch
>
>
> New blocks scanning will be delayed if old blocks deleted after datanode
> restart.
> Steps:
> 1. Write some blocks and wait till all scans over
> 2. Restart the datanode
> 3. Delete some of the blocks
> 4. Write new blocks which are less in size compared to deleted blocks.
> Problem:
> {{BlockPoolSliceScanner#updateBytesToScan(..)}} updates {{bytesLeft}} based
> on following comparison
> {code} if (lastScanTime < currentPeriodStart) {
> bytesLeft += len;
> }{code}
> But in {{BlockPoolSliceScanner#assignInitialVerificationTimes()}}
> {{bytesLeft}} decremented using below comparison
> {code}if (now - entry.verificationTime < scanPeriod) {{code}
> Hence when the old blocks are deleted {{bytesLeft}} going negative.
> new blocks will not be scanned until it becomes positive again.
> So in both places verificationtime should be compared against scanperiod.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6147) New blocks scanning will be delayed due to issue in BlockPoolSliceScanner#updateBytesToScan(..)
[
https://issues.apache.org/jira/browse/HDFS-6147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinayakumar B updated HDFS-6147:
Status: Patch Available (was: Open)
> New blocks scanning will be delayed due to issue in
> BlockPoolSliceScanner#updateBytesToScan(..)
> ---
>
> Key: HDFS-6147
> URL: https://issues.apache.org/jira/browse/HDFS-6147
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.4.0
>Reporter: Vinayakumar B
>Assignee: Vinayakumar B
> Attachments: HDFS-6147.patch
>
>
> New blocks scanning will be delayed if old blocks deleted after datanode
> restart.
> Steps:
> 1. Write some blocks and wait till all scans over
> 2. Restart the datanode
> 3. Delete some of the blocks
> 4. Write new blocks which are less in size compared to deleted blocks.
> Problem:
> {{BlockPoolSliceScanner#updateBytesToScan(..)}} updates {{bytesLeft}} based
> on following comparison
> {code} if (lastScanTime < currentPeriodStart) {
> bytesLeft += len;
> }{code}
> But in {{BlockPoolSliceScanner#assignInitialVerificationTimes()}}
> {{bytesLeft}} decremented using below comparison
> {code}if (now - entry.verificationTime < scanPeriod) {{code}
> Hence when the old blocks are deleted {{bytesLeft}} going negative.
> new blocks will not be scanned until it becomes positive again.
> So in both places verificationtime should be compared against scanperiod.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6147) New blocks scanning will be delayed due to issue in BlockPoolSliceScanner#updateBytesToScan(..)
[
https://issues.apache.org/jira/browse/HDFS-6147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinayakumar B updated HDFS-6147:
Attachment: HDFS-6147.patch
> New blocks scanning will be delayed due to issue in
> BlockPoolSliceScanner#updateBytesToScan(..)
> ---
>
> Key: HDFS-6147
> URL: https://issues.apache.org/jira/browse/HDFS-6147
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.4.0
>Reporter: Vinayakumar B
>Assignee: Vinayakumar B
> Attachments: HDFS-6147.patch
>
>
> New blocks scanning will be delayed if old blocks deleted after datanode
> restart.
> Steps:
> 1. Write some blocks and wait till all scans over
> 2. Restart the datanode
> 3. Delete some of the blocks
> 4. Write new blocks which are less in size compared to deleted blocks.
> Problem:
> {{BlockPoolSliceScanner#updateBytesToScan(..)}} updates {{bytesLeft}} based
> on following comparison
> {code} if (lastScanTime < currentPeriodStart) {
> bytesLeft += len;
> }{code}
> But in {{BlockPoolSliceScanner#assignInitialVerificationTimes()}}
> {{bytesLeft}} decremented using below comparison
> {code}if (now - entry.verificationTime < scanPeriod) {{code}
> Hence when the old blocks are deleted {{bytesLeft}} going negative.
> new blocks will not be scanned until it becomes positive again.
> So in both places verificationtime should be compared against scanperiod.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
