[jira] [Commented] (HDFS-8494) Remove hard-coded chunk size in favor of ECZone
[
https://issues.apache.org/jira/browse/HDFS-8494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14581847#comment-14581847
]
Walter Su commented on HDFS-8494:
-
Zhe Zhang, Kai Zheng and I had a offline discussion. Here is what we thought:
A. Add a cellSize field in BlockInfoStriped as a workaround, and deal with
memory usage in follow-on, only if there is no trouble in FsIamge.
B. If there is trouble about FsImage, we need to figure out a final solution.
bq. we should consider adding a chunk size field to StripedBlockProto and
removing the cell size field from HdfsFileStatus.
I think it means it's ok with approach A, right?. Let's do that. I created
HDFS-8580.
Remove hard-coded chunk size in favor of ECZone
---
Key: HDFS-8494
URL: https://issues.apache.org/jira/browse/HDFS-8494
Project: Hadoop HDFS
Issue Type: Sub-task
Affects Versions: HDFS-7285
Reporter: Kai Sasaki
Assignee: Kai Sasaki
Fix For: HDFS-7285
Attachments: HDFS-8494-HDFS-7285-01.patch,
HDFS-8494-HDFS-7285-02.patch
It is necessary to remove hard-coded values inside NameNode configured in
{{HdfsConstants}}. In this JIRA, we can remove {{chunkSize}} gracefully in
favor of HDFS-8375.
Because {{cellSize}} is now originally stored only in {{ErasureCodingZone}},
{{BlockInfoStriped}} can receive {{cellSize}} in addition to {{ECSchema}}
when its initialization.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8494) Remove hard-coded chunk size in favor of ECZone
[
https://issues.apache.org/jira/browse/HDFS-8494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14582685#comment-14582685
]
Zhe Zhang commented on HDFS-8494:
-
Based on the above discussion, I think we have agreed to go with approach A at
this stage.
I'm also OK with moving this JIRA to HDFS-8031.
Remove hard-coded chunk size in favor of ECZone
---
Key: HDFS-8494
URL: https://issues.apache.org/jira/browse/HDFS-8494
Project: Hadoop HDFS
Issue Type: Sub-task
Affects Versions: HDFS-7285
Reporter: Kai Sasaki
Assignee: Kai Sasaki
Fix For: HDFS-7285
Attachments: HDFS-8494-HDFS-7285-01.patch,
HDFS-8494-HDFS-7285-02.patch
It is necessary to remove hard-coded values inside NameNode configured in
{{HdfsConstants}}. In this JIRA, we can remove {{chunkSize}} gracefully in
favor of HDFS-8375.
Because {{cellSize}} is now originally stored only in {{ErasureCodingZone}},
{{BlockInfoStriped}} can receive {{cellSize}} in addition to {{ECSchema}}
when its initialization.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8494) Remove hard-coded chunk size in favor of ECZone
[
https://issues.apache.org/jira/browse/HDFS-8494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14577615#comment-14577615
]
Jing Zhao commented on HDFS-8494:
-
Also, we should consider adding a chunk size field to {{StripedBlockProto}} and
removing the cell size field from HdfsFileStatus. In this way we can access the
chunk size information in the storage layer.
Remove hard-coded chunk size in favor of ECZone
---
Key: HDFS-8494
URL: https://issues.apache.org/jira/browse/HDFS-8494
Project: Hadoop HDFS
Issue Type: Sub-task
Affects Versions: HDFS-7285
Reporter: Kai Sasaki
Assignee: Kai Sasaki
Fix For: HDFS-7285
Attachments: HDFS-8494-HDFS-7285-01.patch,
HDFS-8494-HDFS-7285-02.patch
It is necessary to remove hard-coded values inside NameNode configured in
{{HdfsConstants}}. In this JIRA, we can remove {{chunkSize}} gracefully in
favor of HDFS-8375.
Because {{cellSize}} is now originally stored only in {{ErasureCodingZone}},
{{BlockInfoStriped}} can receive {{cellSize}} in addition to {{ECSchema}}
when its initialization.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8494) Remove hard-coded chunk size in favor of ECZone
[
https://issues.apache.org/jira/browse/HDFS-8494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14577558#comment-14577558
]
Zhe Zhang commented on HDFS-8494:
-
We can also consider creating a data structure with both EC schema and
{{cellSize}} to represent _EC policy_ for the file. Something like
{{ErasureCodingZone}}, but minus the {{dir}}. [~vinayrpet] I think we discussed
this under HDFS-8375 and HDFS-8408. What's your take on this?
Remove hard-coded chunk size in favor of ECZone
---
Key: HDFS-8494
URL: https://issues.apache.org/jira/browse/HDFS-8494
Project: Hadoop HDFS
Issue Type: Sub-task
Affects Versions: HDFS-7285
Reporter: Kai Sasaki
Assignee: Kai Sasaki
Fix For: HDFS-7285
Attachments: HDFS-8494-HDFS-7285-01.patch,
HDFS-8494-HDFS-7285-02.patch
It is necessary to remove hard-coded values inside NameNode configured in
{{HdfsConstants}}. In this JIRA, we can remove {{chunkSize}} gracefully in
favor of HDFS-8375.
Because {{cellSize}} is now originally stored only in {{ErasureCodingZone}},
{{BlockInfoStriped}} can receive {{cellSize}} in addition to {{ECSchema}}
when its initialization.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8494) Remove hard-coded chunk size in favor of ECZone
[
https://issues.apache.org/jira/browse/HDFS-8494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14577586#comment-14577586
]
Jing Zhao commented on HDFS-8494:
-
Thanks for working on this, [~kaisasak]!
In general I'm not very sure if we need to remove the hard-coded chunk size at
this stage. But if we want to do this, we should record this information in
BlockInfoStriped so that BlockInfoStriped is self explained. I think the basic
idea is that we'd better decouple the logic of the blocks, which belong to the
storage level, from the files. This means if one day we separate the block
management out of the current namenode, the block manager service can provide
enough information for a reader to read the striped block.
For memory footprint, we can easily do some follow-on optimization to save this
extra 4 bytes. E.g., we can store all the existing combinations of (ecschema +
chunk size) somewhere and only record an id in each striped block.
Remove hard-coded chunk size in favor of ECZone
---
Key: HDFS-8494
URL: https://issues.apache.org/jira/browse/HDFS-8494
Project: Hadoop HDFS
Issue Type: Sub-task
Affects Versions: HDFS-7285
Reporter: Kai Sasaki
Assignee: Kai Sasaki
Fix For: HDFS-7285
Attachments: HDFS-8494-HDFS-7285-01.patch,
HDFS-8494-HDFS-7285-02.patch
It is necessary to remove hard-coded values inside NameNode configured in
{{HdfsConstants}}. In this JIRA, we can remove {{chunkSize}} gracefully in
favor of HDFS-8375.
Because {{cellSize}} is now originally stored only in {{ErasureCodingZone}},
{{BlockInfoStriped}} can receive {{cellSize}} in addition to {{ECSchema}}
when its initialization.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8494) Remove hard-coded chunk size in favor of ECZone
[
https://issues.apache.org/jira/browse/HDFS-8494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14577607#comment-14577607
]
Zhe Zhang commented on HDFS-8494:
-
Thanks Jing for sharing the thoughts. It's a good idea to store all EC policies
and refer to the IDs. Looks like this can be done in
{{ErasureCodingZoneManager}}, similar to how encryption zones are stored in a
map.
Remove hard-coded chunk size in favor of ECZone
---
Key: HDFS-8494
URL: https://issues.apache.org/jira/browse/HDFS-8494
Project: Hadoop HDFS
Issue Type: Sub-task
Affects Versions: HDFS-7285
Reporter: Kai Sasaki
Assignee: Kai Sasaki
Fix For: HDFS-7285
Attachments: HDFS-8494-HDFS-7285-01.patch,
HDFS-8494-HDFS-7285-02.patch
It is necessary to remove hard-coded values inside NameNode configured in
{{HdfsConstants}}. In this JIRA, we can remove {{chunkSize}} gracefully in
favor of HDFS-8375.
Because {{cellSize}} is now originally stored only in {{ErasureCodingZone}},
{{BlockInfoStriped}} can receive {{cellSize}} in addition to {{ECSchema}}
when its initialization.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8494) Remove hard-coded chunk size in favor of ECZone
[
https://issues.apache.org/jira/browse/HDFS-8494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14577545#comment-14577545
]
Zhe Zhang commented on HDFS-8494:
-
Thanks for the discussion folks.
I remember discussing this in an offline meetup a while ago with [~jingzhao]
and [~szetszwo]. The conclusion at that time was that we should avoid storing
EC schema for each block.
But I think the question is worth discussing again. A medium size cluster could
have several 100M blocks. 4b a block translates to a few GB of memory overhead.
On the other side of the tradeoff, as Walter analyzed, we need to go through a
few hops to lookup the {{cellSize}}. The most expensive part is probably the
path traversal and {{getECZone}}. So if we don't need to get {{cellSize}}
frequently, I'd suggest keeping in in the zone. We can certainly add a public
method in {{BlockInfoStriped}} to simplify the code.
Remove hard-coded chunk size in favor of ECZone
---
Key: HDFS-8494
URL: https://issues.apache.org/jira/browse/HDFS-8494
Project: Hadoop HDFS
Issue Type: Sub-task
Affects Versions: HDFS-7285
Reporter: Kai Sasaki
Assignee: Kai Sasaki
Fix For: HDFS-7285
Attachments: HDFS-8494-HDFS-7285-01.patch,
HDFS-8494-HDFS-7285-02.patch
It is necessary to remove hard-coded values inside NameNode configured in
{{HdfsConstants}}. In this JIRA, we can remove {{chunkSize}} gracefully in
favor of HDFS-8375.
Because {{cellSize}} is now originally stored only in {{ErasureCodingZone}},
{{BlockInfoStriped}} can receive {{cellSize}} in addition to {{ECSchema}}
when its initialization.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8494) Remove hard-coded chunk size in favor of ECZone
[
https://issues.apache.org/jira/browse/HDFS-8494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14578291#comment-14578291
]
Kai Sasaki commented on HDFS-8494:
--
Thanks all for sharing inspiring ideas and thoughts! I think removing
hard-coded chunk-size requires more discussion and seems to have some
dependencies to solve.
So in this stage, we can assume chunk sizes used in EC cluster are the same to
{{HdfsConstants}} one.
{quote}
I think the basic idea is that we'd better decouple the logic of the blocks,
which belong to the storage level, from the files.
{quote}
I agree with [~jingzhao]. In order to remove hard-coded value, it might be
necessary to discuss more about the self-explained data structure in all cases.
Can we move this JIRA to under HDFS-8030 or HDFS-8031?
Remove hard-coded chunk size in favor of ECZone
---
Key: HDFS-8494
URL: https://issues.apache.org/jira/browse/HDFS-8494
Project: Hadoop HDFS
Issue Type: Sub-task
Affects Versions: HDFS-7285
Reporter: Kai Sasaki
Assignee: Kai Sasaki
Fix For: HDFS-7285
Attachments: HDFS-8494-HDFS-7285-01.patch,
HDFS-8494-HDFS-7285-02.patch
It is necessary to remove hard-coded values inside NameNode configured in
{{HdfsConstants}}. In this JIRA, we can remove {{chunkSize}} gracefully in
favor of HDFS-8375.
Because {{cellSize}} is now originally stored only in {{ErasureCodingZone}},
{{BlockInfoStriped}} can receive {{cellSize}} in addition to {{ECSchema}}
when its initialization.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8494) Remove hard-coded chunk size in favor of ECZone
[
https://issues.apache.org/jira/browse/HDFS-8494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14576683#comment-14576683
]
Kai Zheng commented on HDFS-8494:
-
Comments for the latest patch:
1. To avoid the overhead of {{getErasureCodingZone}} for each
{{StripedDataStreamer}} in the following change, please pass
{{ErasureCodingZone}} in the constructor of StripedDataStreamer, sharing the
same instance of ErasureCodingZone for all the needed streamers.
{code}
void putLoactedBlocks(LocatedBlock lb) throws IOException {
+ErasureCodingZone ecZone = dfsClient.getErasureCodingZone(src);
if (LOG.isDebugEnabled()) {
LOG.debug(Obtained block group + lb);
}
LocatedBlock[] blocks = StripedBlockUtil.parseStripedBlockGroup(
(LocatedStripedBlock)lb,
-BLOCK_STRIPED_CELL_SIZE, NUM_DATA_BLOCKS, NUM_PARITY_BLOCKS);
+ecZone.getCellSize(), NUM_DATA_BLOCKS, NUM_PARITY_BLOCKS);
{code}
2. In balancer/Dispatcher, need to investigate how to remove the hard-coded
value. I would open separate JIRA for this as follow-on.
{code}
- HdfsConstants.BLOCK_STRIPED_CELL_SIZE, dataBlockNum, idxInGroup);
+ HdfsConstants.DEFAULT_BLOCK_STRIPED_CELL_SIZE, dataBlockNum,
idxInGroup);
{code}
3. Please avoid the 2nd call to {{getCellSize}}.
{code}
+ int cellSize = getCellSize(sblock);
+ if (cellSize 0) {
+dataBlockNum = (short) Math.min(dataBlockNum,
+(sblock.getNumBytes() - 1) / getCellSize(sblock) + 1);
+ }
{code}
Remove hard-coded chunk size in favor of ECZone
---
Key: HDFS-8494
URL: https://issues.apache.org/jira/browse/HDFS-8494
Project: Hadoop HDFS
Issue Type: Sub-task
Affects Versions: HDFS-7285
Reporter: Kai Sasaki
Assignee: Kai Sasaki
Fix For: HDFS-7285
Attachments: HDFS-8494-HDFS-7285-01.patch,
HDFS-8494-HDFS-7285-02.patch
It is necessary to remove hard-coded values inside NameNode configured in
{{HdfsConstants}}. In this JIRA, we can remove {{chunkSize}} gracefully in
favor of HDFS-8375.
Because {{cellSize}} is now originally stored only in {{ErasureCodingZone}},
{{BlockInfoStriped}} can receive {{cellSize}} in addition to {{ECSchema}}
when its initialization.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8494) Remove hard-coded chunk size in favor of ECZone
[
https://issues.apache.org/jira/browse/HDFS-8494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14576708#comment-14576708
]
Walter Su commented on HDFS-8494:
-
bq. In balancer/Dispatcher, need to investigate how to remove the hard-coded
value.
It's easy to pass cellSize from BlockInfoStriped to Dispatcher. But the
question is BlockInfoStriped doesn't have it.
You have to get it by: BlockInfoStriped -- getBlockCollection -- getZone --
getCellSize. A bit complicated, isn't it?
I think BlockInfoStriped needs to keep cellSize.
Remove hard-coded chunk size in favor of ECZone
---
Key: HDFS-8494
URL: https://issues.apache.org/jira/browse/HDFS-8494
Project: Hadoop HDFS
Issue Type: Sub-task
Affects Versions: HDFS-7285
Reporter: Kai Sasaki
Assignee: Kai Sasaki
Fix For: HDFS-7285
Attachments: HDFS-8494-HDFS-7285-01.patch,
HDFS-8494-HDFS-7285-02.patch
It is necessary to remove hard-coded values inside NameNode configured in
{{HdfsConstants}}. In this JIRA, we can remove {{chunkSize}} gracefully in
favor of HDFS-8375.
Because {{cellSize}} is now originally stored only in {{ErasureCodingZone}},
{{BlockInfoStriped}} can receive {{cellSize}} in addition to {{ECSchema}}
when its initialization.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8494) Remove hard-coded chunk size in favor of ECZone
[
https://issues.apache.org/jira/browse/HDFS-8494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14577013#comment-14577013
]
Vinayakumar B commented on HDFS-8494:
-
bq. I think BlockInfoStriped needs to keep cellSize.
I too was thinking the same when the FSImageLoader problem has came up. This
will increase the memory usage by ~4bytes for each block though.
Remove hard-coded chunk size in favor of ECZone
---
Key: HDFS-8494
URL: https://issues.apache.org/jira/browse/HDFS-8494
Project: Hadoop HDFS
Issue Type: Sub-task
Affects Versions: HDFS-7285
Reporter: Kai Sasaki
Assignee: Kai Sasaki
Fix For: HDFS-7285
Attachments: HDFS-8494-HDFS-7285-01.patch,
HDFS-8494-HDFS-7285-02.patch
It is necessary to remove hard-coded values inside NameNode configured in
{{HdfsConstants}}. In this JIRA, we can remove {{chunkSize}} gracefully in
favor of HDFS-8375.
Because {{cellSize}} is now originally stored only in {{ErasureCodingZone}},
{{BlockInfoStriped}} can receive {{cellSize}} in addition to {{ECSchema}}
when its initialization.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8494) Remove hard-coded chunk size in favor of ECZone
[
https://issues.apache.org/jira/browse/HDFS-8494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14577096#comment-14577096
]
Kai Zheng commented on HDFS-8494:
-
Let's discuss the case in HDFS-8558 separately to make it easy here. Thanks!
Remove hard-coded chunk size in favor of ECZone
---
Key: HDFS-8494
URL: https://issues.apache.org/jira/browse/HDFS-8494
Project: Hadoop HDFS
Issue Type: Sub-task
Affects Versions: HDFS-7285
Reporter: Kai Sasaki
Assignee: Kai Sasaki
Fix For: HDFS-7285
Attachments: HDFS-8494-HDFS-7285-01.patch,
HDFS-8494-HDFS-7285-02.patch
It is necessary to remove hard-coded values inside NameNode configured in
{{HdfsConstants}}. In this JIRA, we can remove {{chunkSize}} gracefully in
favor of HDFS-8375.
Because {{cellSize}} is now originally stored only in {{ErasureCodingZone}},
{{BlockInfoStriped}} can receive {{cellSize}} in addition to {{ECSchema}}
when its initialization.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8494) Remove hard-coded chunk size in favor of ECZone
[
https://issues.apache.org/jira/browse/HDFS-8494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14574523#comment-14574523
]
Hadoop QA commented on HDFS-8494:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 18m 32s | Pre-patch HDFS-7285 has 5
extant Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 15 new or modified test files. |
| {color:green}+1{color} | javac | 7m 32s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 40s | There were no new javadoc
warning messages. |
| {color:red}-1{color} | release audit | 0m 15s | The applied patch generated
1 release audit warnings. |
| {color:green}+1{color} | checkstyle | 1m 56s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 3s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 37s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 4m 10s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 13s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 172m 36s | Tests failed in hadoop-hdfs. |
| {color:green}+1{color} | hdfs tests | 0m 16s | Tests passed in
hadoop-hdfs-client. |
| | | 220m 27s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.hdfs.TestEncryptedTransfer |
| | hadoop.hdfs.server.namenode.TestFSEditLogLoader |
| | hadoop.hdfs.TestDFSStripedInputStream |
| | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS |
| |
hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerWithStripedBlocks |
| | hadoop.hdfs.TestRecoverStripedFile |
| | hadoop.hdfs.server.namenode.TestAuditLogs |
| | hadoop.hdfs.server.namenode.TestFsck |
| | hadoop.hdfs.server.namenode.TestStripedINodeFile |
| | hadoop.hdfs.server.namenode.TestFSImage |
| | hadoop.hdfs.server.namenode.TestAddStripedBlocks |
| | hadoop.hdfs.server.blockmanagement.TestBlockInfo |
| | hadoop.hdfs.server.namenode.TestFileTruncate |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12737906/HDFS-8494-HDFS-7285-02.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | HDFS-7285 / c0929ab |
| Pre-patch Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/11239/artifact/patchprocess/HDFS-7285FindbugsWarningshadoop-hdfs.html
|
| Pre-patch Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/11239/artifact/patchprocess/HDFS-7285FindbugsWarningshadoop-hdfs-client.html
|
| Release Audit |
https://builds.apache.org/job/PreCommit-HDFS-Build/11239/artifact/patchprocess/patchReleaseAuditProblems.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/11239/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| hadoop-hdfs-client test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/11239/artifact/patchprocess/testrun_hadoop-hdfs-client.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/11239/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/11239/console |
This message was automatically generated.
Remove hard-coded chunk size in favor of ECZone
---
Key: HDFS-8494
URL: https://issues.apache.org/jira/browse/HDFS-8494
Project: Hadoop HDFS
Issue Type: Sub-task
Affects Versions: HDFS-7285
Reporter: Kai Sasaki
Assignee: Kai Sasaki
Fix For: HDFS-7285
Attachments: HDFS-8494-HDFS-7285-01.patch,
HDFS-8494-HDFS-7285-02.patch
It is necessary to remove hard-coded values inside NameNode configured in
{{HdfsConstants}}. In this JIRA, we can remove {{chunkSize}} gracefully in
favor of HDFS-8375.
Because {{cellSize}} is now originally stored only in {{ErasureCodingZone}},
{{BlockInfoStriped}} can receive {{cellSize}} in addition to {{ECSchema}}
when its initialization.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8494) Remove hard-coded chunk size in favor of ECZone
[
https://issues.apache.org/jira/browse/HDFS-8494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14570638#comment-14570638
]
Vinayakumar B commented on HDFS-8494:
-
bq. I just modified the description as it's obsolete. In the patch it passed
cellSize along with the schema to initialize a BlockInfoStriped. I thought it
works, right?
Still the patch doesnt pass cellSize for the inititalize. It passes only for
spaceConsumed(). I think its fine as of now.
Need to find the way to optimize further
Remove hard-coded chunk size in favor of ECZone
---
Key: HDFS-8494
URL: https://issues.apache.org/jira/browse/HDFS-8494
Project: Hadoop HDFS
Issue Type: Sub-task
Affects Versions: HDFS-7285
Reporter: Kai Sasaki
Assignee: Kai Sasaki
Fix For: HDFS-7285
Attachments: HDFS-8494-HDFS-7285-01.patch
It is necessary to remove hard-coded values inside NameNode configured in
{{HdfsConstants}}. In this JIRA, we can remove {{chunkSize}} gracefully in
favor of HDFS-8375.
Because {{cellSize}} is now originally stored only in {{ErasureCodingZone}},
{{BlockInfoStriped}} can receive {{cellSize}} in addition to {{ECSchema}}
when its initialization.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8494) Remove hard-coded chunk size in favor of ECZone
[
https://issues.apache.org/jira/browse/HDFS-8494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14570552#comment-14570552
]
Kai Zheng commented on HDFS-8494:
-
bq. as mentioned in description, why ecZone itself not passed instead of
ecschema during initialization of patch?
I just modified the description as it's obsolete. In the patch it passed
cellSize along with the schema to initialize a BlockInfoStriped. I thought it
works, right?
Remove hard-coded chunk size in favor of ECZone
---
Key: HDFS-8494
URL: https://issues.apache.org/jira/browse/HDFS-8494
Project: Hadoop HDFS
Issue Type: Sub-task
Affects Versions: HDFS-7285
Reporter: Kai Sasaki
Assignee: Kai Sasaki
Fix For: HDFS-7285
Attachments: HDFS-8494-HDFS-7285-01.patch
It is necessary to remove hard-coded values inside NameNode configured in
{{HdfsConstants}}. In this JIRA, we can remove {{chunkSize}} gracefully in
favor of HDFS-8375.
Because {{cellSize}} is now originally stored only in {{ErasureCodingZone}},
{{BlockInfoStriped}} can receive {{cellSize}} in addition to {{ECSchema}}
when its initialization.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8494) Remove hard-coded chunk size in favor of ECZone
[
https://issues.apache.org/jira/browse/HDFS-8494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14570544#comment-14570544
]
Vinayakumar B commented on HDFS-8494:
-
I didnt review the patch fully, but based on discussion with [~drankye],
I think {{BlockInfoStriped#spaceConsumed()}} is not called while loading from
fsimage, instead it might be called while loading edits. But at this time, we
can get the zone.
One thing I got at the first look of the patch,
as mentioned in description, why ecZone itself not passed instead of ecschema
during initialization of patch?
I think this would make things clear. No need to worry about whether zone
information available during image loading or edits loading to know the cellSize
Remove hard-coded chunk size in favor of ECZone
---
Key: HDFS-8494
URL: https://issues.apache.org/jira/browse/HDFS-8494
Project: Hadoop HDFS
Issue Type: Sub-task
Affects Versions: HDFS-7285
Reporter: Kai Sasaki
Assignee: Kai Sasaki
Fix For: HDFS-7285
Attachments: HDFS-8494-HDFS-7285-01.patch
It is necessary to remove hard-coded values inside NameNode configured in
{{HdfsConstants}}. In this JIRA, we can remove {{chunkSize}} gracefully in
favor of HDFS-8375.
Because {{cellSize}} is now originally stored only in {{ErasureCodingZone}},
{{BlockInfoStriped}} can receive {{ErasureCodingZone}} instead of
{{ECSchema}} when its initialization.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8494) Remove hard-coded chunk size in favor of ECZone
[
https://issues.apache.org/jira/browse/HDFS-8494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14570548#comment-14570548
]
Kai Zheng commented on HDFS-8494:
-
Just had an off-line discussion with Vinay about the above question and sort
something out. We should avoid relying on cellSize during image loading phase.
For the code in question (used by getFileStatus), it's actually used in two
places: FSImageHandler for Web HDFS, and a test
TestOfflineImageViewerWithStripedBlocks. For the first, we can get the zone
info anyway since it's not really called during startup; the second, we can
change the test to pass the cellSize value (used by the test).
Thanks Vinay for the discussion. Kai please let know if it works or not.
Remove hard-coded chunk size in favor of ECZone
---
Key: HDFS-8494
URL: https://issues.apache.org/jira/browse/HDFS-8494
Project: Hadoop HDFS
Issue Type: Sub-task
Affects Versions: HDFS-7285
Reporter: Kai Sasaki
Assignee: Kai Sasaki
Fix For: HDFS-7285
Attachments: HDFS-8494-HDFS-7285-01.patch
It is necessary to remove hard-coded values inside NameNode configured in
{{HdfsConstants}}. In this JIRA, we can remove {{chunkSize}} gracefully in
favor of HDFS-8375.
Because {{cellSize}} is now originally stored only in {{ErasureCodingZone}},
{{BlockInfoStriped}} can receive {{ErasureCodingZone}} instead of
{{ECSchema}} when its initialization.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8494) Remove hard-coded chunk size in favor of ECZone
[
https://issues.apache.org/jira/browse/HDFS-8494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14572075#comment-14572075
]
Kai Sasaki commented on HDFS-8494:
--
[~drankye] Thanks quick response. I make a separate JIRA in order to resolve
issue in {{FSImageLoader}}.
We are working only removing chunk size other than {{FSImageLoader}}. Thank you.
Remove hard-coded chunk size in favor of ECZone
---
Key: HDFS-8494
URL: https://issues.apache.org/jira/browse/HDFS-8494
Project: Hadoop HDFS
Issue Type: Sub-task
Affects Versions: HDFS-7285
Reporter: Kai Sasaki
Assignee: Kai Sasaki
Fix For: HDFS-7285
Attachments: HDFS-8494-HDFS-7285-01.patch
It is necessary to remove hard-coded values inside NameNode configured in
{{HdfsConstants}}. In this JIRA, we can remove {{chunkSize}} gracefully in
favor of HDFS-8375.
Because {{cellSize}} is now originally stored only in {{ErasureCodingZone}},
{{BlockInfoStriped}} can receive {{cellSize}} in addition to {{ECSchema}}
when its initialization.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8494) Remove hard-coded chunk size in favor of ECZone
[
https://issues.apache.org/jira/browse/HDFS-8494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14571956#comment-14571956
]
Kai Sasaki commented on HDFS-8494:
--
Thank you so much for clarifying critical points.
{quote}
FSImageHandler for Web HDFS, ...
... For the first, we can get the zone info anyway since it's not really called
during startup
{quote}
I cannot fully understand this point. {{FSImageHandler}} is used by Web HDFS,
{{WebImageViewer}}. {{WebImageViewer}} looks keep only fsimage data. This data
is restored through {{FSImageLoader}}. So anyway is it necessary to restore
ECZone info from fsimage file? Is there other way we should restore ECZone info
from NN or something?
Remove hard-coded chunk size in favor of ECZone
---
Key: HDFS-8494
URL: https://issues.apache.org/jira/browse/HDFS-8494
Project: Hadoop HDFS
Issue Type: Sub-task
Affects Versions: HDFS-7285
Reporter: Kai Sasaki
Assignee: Kai Sasaki
Fix For: HDFS-7285
Attachments: HDFS-8494-HDFS-7285-01.patch
It is necessary to remove hard-coded values inside NameNode configured in
{{HdfsConstants}}. In this JIRA, we can remove {{chunkSize}} gracefully in
favor of HDFS-8375.
Because {{cellSize}} is now originally stored only in {{ErasureCodingZone}},
{{BlockInfoStriped}} can receive {{cellSize}} in addition to {{ECSchema}}
when its initialization.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8494) Remove hard-coded chunk size in favor of ECZone
[
https://issues.apache.org/jira/browse/HDFS-8494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14572063#comment-14572063
]
Kai Zheng commented on HDFS-8494:
-
I thought this may be a hard case that can be resolved separately. This patch
is already big. Kai Sasaki would you separate it and move on this? Thanks.
Remove hard-coded chunk size in favor of ECZone
---
Key: HDFS-8494
URL: https://issues.apache.org/jira/browse/HDFS-8494
Project: Hadoop HDFS
Issue Type: Sub-task
Affects Versions: HDFS-7285
Reporter: Kai Sasaki
Assignee: Kai Sasaki
Fix For: HDFS-7285
Attachments: HDFS-8494-HDFS-7285-01.patch
It is necessary to remove hard-coded values inside NameNode configured in
{{HdfsConstants}}. In this JIRA, we can remove {{chunkSize}} gracefully in
favor of HDFS-8375.
Because {{cellSize}} is now originally stored only in {{ErasureCodingZone}},
{{BlockInfoStriped}} can receive {{cellSize}} in addition to {{ECSchema}}
when its initialization.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8494) Remove hard-coded chunk size in favor of ECZone
[
https://issues.apache.org/jira/browse/HDFS-8494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14572019#comment-14572019
]
Kai Zheng commented on HDFS-8494:
-
I'm not familiar about this. [~zhz] do you have some idea?
Remove hard-coded chunk size in favor of ECZone
---
Key: HDFS-8494
URL: https://issues.apache.org/jira/browse/HDFS-8494
Project: Hadoop HDFS
Issue Type: Sub-task
Affects Versions: HDFS-7285
Reporter: Kai Sasaki
Assignee: Kai Sasaki
Fix For: HDFS-7285
Attachments: HDFS-8494-HDFS-7285-01.patch
It is necessary to remove hard-coded values inside NameNode configured in
{{HdfsConstants}}. In this JIRA, we can remove {{chunkSize}} gracefully in
favor of HDFS-8375.
Because {{cellSize}} is now originally stored only in {{ErasureCodingZone}},
{{BlockInfoStriped}} can receive {{cellSize}} in addition to {{ECSchema}}
when its initialization.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8494) Remove hard-coded chunk size in favor of ECZone
[
https://issues.apache.org/jira/browse/HDFS-8494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14570027#comment-14570027
]
Kai Zheng commented on HDFS-8494:
-
Thanks Kai for the hard work. The patch looks good. Some comments:
1. Maybe we could move the exception handling to {{getCellSize}} so the
multiple callings of it can be simplified, and the message can be more relevant.
{code}
+ int cellSize;
+ try {
+cellSize = getCellSize(stripedBlock);
+ } catch (IOException e) {
+blockLog
+.warn(Failed to get the EC zone for the file {} ,
+stripedBlock.getBlockCollection().getName());
+return null;
+ }
{code}
2. In the following change: 1) the TODO isn't correct, because cellSize and
ECZone are from XAttr, just remove it; 2) why we use the default constant?
{code}
+// TODO: Cell size and ECZone info should be restored from
FSImage(HDFS-7859).
size += StripedBlockUtil.spaceConsumedByStripedBlock(p.getBlock()
.getNumBytes(), p.getDataBlockNum(), p.getParityBlockNum(),
-HdfsConstants.BLOCK_STRIPED_CELL_SIZE);
+HdfsConstants.DEFAULT_BLOCK_STRIPED_CELL_SIZE);
{code}
Remove hard-coded chunk size in favor of ECZone
---
Key: HDFS-8494
URL: https://issues.apache.org/jira/browse/HDFS-8494
Project: Hadoop HDFS
Issue Type: Sub-task
Affects Versions: HDFS-7285
Reporter: Kai Sasaki
Assignee: Kai Sasaki
Fix For: HDFS-7285
Attachments: HDFS-8494-HDFS-7285-01.patch
It is necessary to remove hard-coded values inside NameNode configured in
{{HdfsConstants}}. In this JIRA, we can remove {{chunkSize}} gracefully in
favor of HDFS-8375.
Because {{cellSize}} is now originally stored only in {{ErasureCodingZone}},
{{BlockInfoStriped}} can receive {{ErasureCodingZone}} instead of
{{ECSchema}} when its initialization.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8494) Remove hard-coded chunk size in favor of ECZone
[
https://issues.apache.org/jira/browse/HDFS-8494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14570123#comment-14570123
]
Kai Sasaki commented on HDFS-8494:
--
[~drankye] Thank you for reviewing. I can update 1. However I have a question
about point 2. Although we can restore cell size from ECZone, is it possible to
restore ECZone when restoring file system from fsimage file? In this case
inside {{FSImageLoader}} we seem to be unable to restore ECZone information
with current code.
Remove hard-coded chunk size in favor of ECZone
---
Key: HDFS-8494
URL: https://issues.apache.org/jira/browse/HDFS-8494
Project: Hadoop HDFS
Issue Type: Sub-task
Affects Versions: HDFS-7285
Reporter: Kai Sasaki
Assignee: Kai Sasaki
Fix For: HDFS-7285
Attachments: HDFS-8494-HDFS-7285-01.patch
It is necessary to remove hard-coded values inside NameNode configured in
{{HdfsConstants}}. In this JIRA, we can remove {{chunkSize}} gracefully in
favor of HDFS-8375.
Because {{cellSize}} is now originally stored only in {{ErasureCodingZone}},
{{BlockInfoStriped}} can receive {{ErasureCodingZone}} instead of
{{ECSchema}} when its initialization.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8494) Remove hard-coded chunk size in favor of ECZone
[
https://issues.apache.org/jira/browse/HDFS-8494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14570196#comment-14570196
]
Kai Zheng commented on HDFS-8494:
-
bq. is it possible to restore ECZone when restoring file system from fsimage
file?
Good question. [~vinayrpet] do you have some idea? Thanks.
Remove hard-coded chunk size in favor of ECZone
---
Key: HDFS-8494
URL: https://issues.apache.org/jira/browse/HDFS-8494
Project: Hadoop HDFS
Issue Type: Sub-task
Affects Versions: HDFS-7285
Reporter: Kai Sasaki
Assignee: Kai Sasaki
Fix For: HDFS-7285
Attachments: HDFS-8494-HDFS-7285-01.patch
It is necessary to remove hard-coded values inside NameNode configured in
{{HdfsConstants}}. In this JIRA, we can remove {{chunkSize}} gracefully in
favor of HDFS-8375.
Because {{cellSize}} is now originally stored only in {{ErasureCodingZone}},
{{BlockInfoStriped}} can receive {{ErasureCodingZone}} instead of
{{ECSchema}} when its initialization.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8494) Remove hard-coded chunk size in favor of ECZone
[
https://issues.apache.org/jira/browse/HDFS-8494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14569418#comment-14569418
]
Hadoop QA commented on HDFS-8494:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 15m 45s | Findbugs (version ) appears to
be broken on HDFS-7285. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 13 new or modified test files. |
| {color:green}+1{color} | javac | 7m 45s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 47s | There were no new javadoc
warning messages. |
| {color:red}-1{color} | release audit | 0m 13s | The applied patch generated
1 release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 52s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 2s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 42s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 31s | The patch built with
eclipse:eclipse. |
| {color:red}-1{color} | findbugs | 4m 16s | The patch appears to introduce 2
new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 17s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 175m 3s | Tests failed in hadoop-hdfs. |
| {color:green}+1{color} | hdfs tests | 0m 16s | Tests passed in
hadoop-hdfs-client. |
| | | 219m 36s | |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs |
| FindBugs | module:hadoop-hdfs-client |
| Failed unit tests |
hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerWithStripedBlocks |
| | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS |
| | hadoop.hdfs.TestWriteReadStripedFile |
| | hadoop.hdfs.TestRecoverStripedFile |
| | hadoop.hdfs.server.namenode.TestFSEditLogLoader |
| | hadoop.hdfs.TestDFSStripedInputStream |
| | hadoop.hdfs.server.namenode.TestFileTruncate |
| | hadoop.hdfs.server.blockmanagement.TestBlockInfo |
| | hadoop.hdfs.server.namenode.TestAuditLogs |
| | hadoop.hdfs.server.namenode.TestFsck |
| | hadoop.hdfs.server.namenode.TestAddStripedBlocks |
| | hadoop.hdfs.server.namenode.TestStripedINodeFile |
| | hadoop.hdfs.TestEncryptedTransfer |
| | hadoop.hdfs.server.namenode.TestFSImage |
| | hadoop.hdfs.TestRollingUpgrade |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12736825/HDFS-8494-HDFS-7285-01.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | HDFS-7285 / 2d847e7 |
| Release Audit |
https://builds.apache.org/job/PreCommit-HDFS-Build/11203/artifact/patchprocess/patchReleaseAuditProblems.txt
|
| Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/11203/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
|
| Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/11203/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs-client.html
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/11203/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| hadoop-hdfs-client test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/11203/artifact/patchprocess/testrun_hadoop-hdfs-client.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/11203/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf908.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/11203/console |
This message was automatically generated.
Remove hard-coded chunk size in favor of ECZone
---
Key: HDFS-8494
URL: https://issues.apache.org/jira/browse/HDFS-8494
Project: Hadoop HDFS
Issue Type: Sub-task
Affects Versions: HDFS-7285
Reporter: Kai Sasaki
Assignee: Kai Sasaki
Fix For: HDFS-7285
Attachments: HDFS-8494-HDFS-7285-01.patch
It is necessary to remove hard-coded values inside NameNode configured in
{{HdfsConstants}}. In this JIRA, we can remove {{chunkSize}} gracefully in
favor of HDFS-8375.
Because {{cellSize}} is now originally stored only in {{ErasureCodingZone}},
{{BlockInfoStriped}} can receive {{ErasureCodingZone}} instead of
{{ECSchema}} when its initialization.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8494) Remove hard-coded chunk size in favor of ECZone
[
https://issues.apache.org/jira/browse/HDFS-8494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14568374#comment-14568374
]
Kai Zheng commented on HDFS-8494:
-
Yes it looks needed. Would you provide the patch? Thanks.
Remove hard-coded chunk size in favor of ECZone
---
Key: HDFS-8494
URL: https://issues.apache.org/jira/browse/HDFS-8494
Project: Hadoop HDFS
Issue Type: Sub-task
Affects Versions: HDFS-7285
Reporter: Kai Sasaki
Assignee: Kai Sasaki
Fix For: HDFS-7285
It is necessary to remove hard-coded values inside NameNode configured in
{{HdfsConstants}}. In this JIRA, we can remove {{chunkSize}} gracefully in
favor of HDFS-8375.
Because {{cellSize}} is now originally stored only in {{ErasureCodingZone}},
{{BlockInfoStriped}} can receive {{ErasureCodingZone}} instead of
{{ECSchema}} when its initialization.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8494) Remove hard-coded chunk size in favor of ECZone
[
https://issues.apache.org/jira/browse/HDFS-8494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14566818#comment-14566818
]
Kai Zheng commented on HDFS-8494:
-
bq. BlockInfoStriped needs cellSize to calculate consumed space.
Did HDFS-8375 miss the change required here? What's the value that's used in
existing codes?
bq. Is it necessary to pass ErasureCodingZone to BlockInfoStriped?
Please don't. If necessary please just pass {{cellSize}} along with schema as
did in HDFS-8375.
[~vinayrpet], do we have anything to add here regarding the chunkSize/cellSize
stuff here?
I thought we could change {{HdfsConstants.BLOCK_STRIPED_CELL_SIZE}} to
{{DEFAULT_BLOCK_STRIPED_CELL_SIZE}} and ensure it's only used when really
necessary.
Remove hard-coded chunk size in favor of ECZone
---
Key: HDFS-8494
URL: https://issues.apache.org/jira/browse/HDFS-8494
Project: Hadoop HDFS
Issue Type: Sub-task
Affects Versions: HDFS-7285
Reporter: Kai Sasaki
Assignee: Kai Sasaki
Fix For: HDFS-7285
It is necessary to remove hard-coded values inside NameNode configured in
{{HdfsConstants}}. In this JIRA, we can remove {{chunkSize}} gracefully in
favor of HDFS-8375.
Because {{cellSize}} is now originally stored only in {{ErasureCodingZone}},
{{BlockInfoStriped}} can receive {{ErasureCodingZone}} instead of
{{ECSchema}} when its initialization.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8494) Remove hard-coded chunk size in favor of ECZone
[
https://issues.apache.org/jira/browse/HDFS-8494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14566944#comment-14566944
]
Kai Sasaki commented on HDFS-8494:
--
{{BlockInfoStriped#spaceConsumed}} looks need BLOCK_STRIPED_CELL_SIZE here.
Originally, does it need to it in order to calculate consumed space?
{code:java}
public long spaceConsumed() {
// In case striped blocks, total usage by this striped blocks should
// be the total of data blocks and parity blocks because
// `getNumBytes` is the total of actual data block size.
return StripedBlockUtil.spaceConsumedByStripedBlock(getNumBytes(),
this.schema.getNumDataUnits(), this.schema.getNumParityUnits(),
BLOCK_STRIPED_CELL_SIZE);
}
{code}
Remove hard-coded chunk size in favor of ECZone
---
Key: HDFS-8494
URL: https://issues.apache.org/jira/browse/HDFS-8494
Project: Hadoop HDFS
Issue Type: Sub-task
Affects Versions: HDFS-7285
Reporter: Kai Sasaki
Assignee: Kai Sasaki
Fix For: HDFS-7285
It is necessary to remove hard-coded values inside NameNode configured in
{{HdfsConstants}}. In this JIRA, we can remove {{chunkSize}} gracefully in
favor of HDFS-8375.
Because {{cellSize}} is now originally stored only in {{ErasureCodingZone}},
{{BlockInfoStriped}} can receive {{ErasureCodingZone}} instead of
{{ECSchema}} when its initialization.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8494) Remove hard-coded chunk size in favor of ECZone
[
https://issues.apache.org/jira/browse/HDFS-8494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14564182#comment-14564182
]
Kai Sasaki commented on HDFS-8494:
--
Current {{BlockInfoStriped}} does not keep {{ErasureCodingZone}}. On the other
hand, {{BlockInfoStriped}} needs cellSize to calculate consumed space. Is it
necessary to pass {{ErasureCodingZone}} to {{BlockInfoStriped}}? Is this
assumption right?
Remove hard-coded chunk size in favor of ECZone
---
Key: HDFS-8494
URL: https://issues.apache.org/jira/browse/HDFS-8494
Project: Hadoop HDFS
Issue Type: Sub-task
Affects Versions: HDFS-7285
Reporter: Kai Sasaki
Assignee: Kai Sasaki
Fix For: HDFS-7285
It is necessary to remove hard-coded values inside NameNode configured in
{{HdfsConstants}}. In this JIRA, we can remove {{chunkSize}} gracefully in
favor of HDFS-8375.
Because {{cellSize}} is now originally stored only in {{ErasureCodingZone}},
{{BlockInfoStriped}} can receive {{ErasureCodingZone}} instead of
{{ECSchema}} when its initialization.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
