[jira] [Commented] (HDFS-8909) Erasure coding: update BlockInfoContiguousUC and BlockInfoStripedUC to use BlockUnderConstructionFeature
[
https://issues.apache.org/jira/browse/HDFS-8909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14716184#comment-14716184
]
Zhe Zhang commented on HDFS-8909:
-
Thanks Jing for updating the patch and Walter for the final review! After this
I will try another "git rebase" to sync with current trunk. If it turns out to
be hard I'll do "git merge".
> Erasure coding: update BlockInfoContiguousUC and BlockInfoStripedUC to use
> BlockUnderConstructionFeature
>
>
> Key: HDFS-8909
> URL: https://issues.apache.org/jira/browse/HDFS-8909
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: HDFS-7285
>Reporter: Zhe Zhang
>Assignee: Jing Zhao
> Attachments: HDFS-8909-HDFS-7285.003.patch,
> HDFS-8909-HDFS-7285.004.patch, HDFS-8909.000.patch, HDFS-8909.001.patch,
> HDFS-8909.002.patch, HDFS-8909.003.patch
>
>
> HDFS-8801 converts {{BlockInfoUC}} as a feature. We should consolidate
> {{BlockInfoContiguousUC}} and {{BlockInfoStripedUC}} logics to use this
> feature.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8909) Erasure coding: update BlockInfoContiguousUC and BlockInfoStripedUC to use BlockUnderConstructionFeature
[
https://issues.apache.org/jira/browse/HDFS-8909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14716182#comment-14716182
]
Walter Su commented on HDFS-8909:
-
+1 for the last patch. None of the failed tests related.
TestDFSStripedOutputStreamWithFailure is failed because of HDFS-8838. I don't
understand why jenkins skipped it. I'll create a follow-on jira to address that.
> Erasure coding: update BlockInfoContiguousUC and BlockInfoStripedUC to use
> BlockUnderConstructionFeature
>
>
> Key: HDFS-8909
> URL: https://issues.apache.org/jira/browse/HDFS-8909
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: HDFS-7285
>Reporter: Zhe Zhang
>Assignee: Jing Zhao
> Attachments: HDFS-8909-HDFS-7285.003.patch,
> HDFS-8909-HDFS-7285.004.patch, HDFS-8909.000.patch, HDFS-8909.001.patch,
> HDFS-8909.002.patch, HDFS-8909.003.patch
>
>
> HDFS-8801 converts {{BlockInfoUC}} as a feature. We should consolidate
> {{BlockInfoContiguousUC}} and {{BlockInfoStripedUC}} logics to use this
> feature.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8964) Provide max TxId when validating in-progress edit log files
[
https://issues.apache.org/jira/browse/HDFS-8964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang updated HDFS-8964:
Attachment: HDFS-8964.00.patch
Initial patch to demonstrate the idea and trigger Jenkins. Will add a test to
emulate the described read-being-written race condition and verify the fix
addresses it.
> Provide max TxId when validating in-progress edit log files
> ---
>
> Key: HDFS-8964
> URL: https://issues.apache.org/jira/browse/HDFS-8964
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: journal-node, namenode
>Affects Versions: 2.7.1
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Attachments: HDFS-8964.00.patch
>
>
> NN/JN validates in-progress edit log files in multiple scenarios, via
> {{EditLogFile#validateLog}}. The method scans through the edit log file to
> find the last transaction ID.
> However, an in-progress edit log file could be actively written to, which
> creates a race condition and causes incorrect data to be read (and later we
> attempt to interpret the data as ops).
> Currently {{validateLog}} is used in 3 places:
> # NN {{getEditsFromTxid}}
> # JN {{getEditLogManifest}}
> # NN/JN {{recoverUnfinalizedSegments}}
> In the first two scenarios we should provide a maximum TxId to validate in
> the in-progress file. The 3rd scenario won't cause a race condition because
> only non-current in-progress edit log files are validated.
> {{validateLog}} is actually only used with in-progress files, and could use a
> better name and Javadoc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8969) Inconsistent synchronization of FSEditLog.editLogStream;
[
https://issues.apache.org/jira/browse/HDFS-8969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14716171#comment-14716171
]
Brahma Reddy Battula commented on HDFS-8969:
[~anu] thanks for reporting this jira..
can you please handle following one also and change the issue summary..?
{noformat}
Redundant nullcheck of blockInfo, which is known to be non-null in
org.apache.hadoop.hdfs.server.namenode.NamenodeFsck.blockIdCK(String)
Bug type RCN_REDUNDANT_NULLCHECK_OF_NONNULL_VALUE (click for details)
In class org.apache.hadoop.hdfs.server.namenode.NamenodeFsck
In method org.apache.hadoop.hdfs.server.namenode.NamenodeFsck.blockIdCK(String)
Value loaded from blockInfo
Return value of
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.getStoredBlock(Block)
Redundant null check at NamenodeFsck.java:[line 257]
{noformat}
> Inconsistent synchronization of FSEditLog.editLogStream;
>
>
> Key: HDFS-8969
> URL: https://issues.apache.org/jira/browse/HDFS-8969
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: HDFS
>Affects Versions: 2.8.0
>Reporter: Anu Engineer
>Assignee: Anu Engineer
>
> Fix Findbug warning on Synchronization.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8964) Provide max TxId when validating in-progress edit log files
[
https://issues.apache.org/jira/browse/HDFS-8964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang updated HDFS-8964:
Status: Patch Available (was: Open)
> Provide max TxId when validating in-progress edit log files
> ---
>
> Key: HDFS-8964
> URL: https://issues.apache.org/jira/browse/HDFS-8964
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: journal-node, namenode
>Affects Versions: 2.7.1
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
>
> NN/JN validates in-progress edit log files in multiple scenarios, via
> {{EditLogFile#validateLog}}. The method scans through the edit log file to
> find the last transaction ID.
> However, an in-progress edit log file could be actively written to, which
> creates a race condition and causes incorrect data to be read (and later we
> attempt to interpret the data as ops).
> Currently {{validateLog}} is used in 3 places:
> # NN {{getEditsFromTxid}}
> # JN {{getEditLogManifest}}
> # NN/JN {{recoverUnfinalizedSegments}}
> In the first two scenarios we should provide a maximum TxId to validate in
> the in-progress file. The 3rd scenario won't cause a race condition because
> only non-current in-progress edit log files are validated.
> {{validateLog}} is actually only used with in-progress files, and could use a
> better name and Javadoc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8823) Move replication factor into individual blocks
[
https://issues.apache.org/jira/browse/HDFS-8823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14716163#comment-14716163
]
Brahma Reddy Battula commented on HDFS-8823:
It appears that this jira introduces one new findbugs warnings in Jenkins
https://builds.apache.org/job/PreCommit-HDFS-Build/12138/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
{noformat}
Redundant nullcheck of blockInfo, which is known to be non-null in
org.apache.hadoop.hdfs.server.namenode.NamenodeFsck.blockIdCK(String)
Bug type RCN_REDUNDANT_NULLCHECK_OF_NONNULL_VALUE (click for details)
In class org.apache.hadoop.hdfs.server.namenode.NamenodeFsck
In method org.apache.hadoop.hdfs.server.namenode.NamenodeFsck.blockIdCK(String)
Value loaded from blockInfo
Return value of
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.getStoredBlock(Block)
Redundant null check at NamenodeFsck.java:[line 257]
{noformat}
> Move replication factor into individual blocks
> --
>
> Key: HDFS-8823
> URL: https://issues.apache.org/jira/browse/HDFS-8823
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Fix For: 2.8.0
>
> Attachments: HDFS-8823.000.patch, HDFS-8823.001.patch,
> HDFS-8823.002.patch, HDFS-8823.003.patch, HDFS-8823.004.patch,
> HDFS-8823.005.patch, HDFS-8823.006.patch
>
>
> This jira proposes to record the replication factor in the {{BlockInfo}}
> class. The changes have two advantages:
> * Decoupling the namespace and the block management layer. It is a
> prerequisite step to move block management off the heap or to a separate
> process.
> * Increased flexibility on replicating blocks. Currently the replication
> factors of all blocks have to be the same. The replication factors of these
> blocks are equal to the highest replication factor across all snapshots. The
> changes will allow blocks in a file to have different replication factor,
> potentially saving some space.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8932) NPE thrown in NameNode when try to get "TotalSyncCount" metric before editLogStream initialization
[
https://issues.apache.org/jira/browse/HDFS-8932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14716159#comment-14716159
]
Vinayakumar B commented on HDFS-8932:
-
I wonder how {{editLogStream}} inconsistence synchronization findbug was
introduced in this Jira, as there is no change related to synchronization in
the patch.
I feel that was a existing findbug, but shown as this patch touches
{{FSEditlog}}.
> NPE thrown in NameNode when try to get "TotalSyncCount" metric before
> editLogStream initialization
> --
>
> Key: HDFS-8932
> URL: https://issues.apache.org/jira/browse/HDFS-8932
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
> Fix For: 2.8.0
>
> Attachments: HDFS-8932.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8941) DistributedFileSystem listCorruptFileBlocks API should resolve relative path
[
https://issues.apache.org/jira/browse/HDFS-8941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14716137#comment-14716137
]
Rakesh R commented on HDFS-8941:
It seems the test case failures are not related to the patch. Let me try to
trigger the jenkins again for a clean report!
> DistributedFileSystem listCorruptFileBlocks API should resolve relative path
>
>
> Key: HDFS-8941
> URL: https://issues.apache.org/jira/browse/HDFS-8941
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Rakesh R
>Assignee: Rakesh R
> Attachments: HDFS-8941-00.patch
>
>
> Presently {{DFS#listCorruptFileBlocks(path)}} API is not resolving the given
> path relative to the workingDir. This jira is to discuss and provide the
> implementation of the same.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8941) DistributedFileSystem listCorruptFileBlocks API should resolve relative path
[
https://issues.apache.org/jira/browse/HDFS-8941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rakesh R updated HDFS-8941:
---
Status: Open (was: Patch Available)
> DistributedFileSystem listCorruptFileBlocks API should resolve relative path
>
>
> Key: HDFS-8941
> URL: https://issues.apache.org/jira/browse/HDFS-8941
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Rakesh R
>Assignee: Rakesh R
> Attachments: HDFS-8941-00.patch
>
>
> Presently {{DFS#listCorruptFileBlocks(path)}} API is not resolving the given
> path relative to the workingDir. This jira is to discuss and provide the
> implementation of the same.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8941) DistributedFileSystem listCorruptFileBlocks API should resolve relative path
[
https://issues.apache.org/jira/browse/HDFS-8941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rakesh R updated HDFS-8941:
---
Status: Patch Available (was: Open)
> DistributedFileSystem listCorruptFileBlocks API should resolve relative path
>
>
> Key: HDFS-8941
> URL: https://issues.apache.org/jira/browse/HDFS-8941
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Rakesh R
>Assignee: Rakesh R
> Attachments: HDFS-8941-00.patch
>
>
> Presently {{DFS#listCorruptFileBlocks(path)}} API is not resolving the given
> path relative to the workingDir. This jira is to discuss and provide the
> implementation of the same.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8967) Create a BlockManagerLock class to represent the lock used in the BlockManager
[
https://issues.apache.org/jira/browse/HDFS-8967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14716134#comment-14716134
]
Hadoop QA commented on HDFS-8967:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 17m 54s | Pre-patch trunk has 4 extant
Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 3 new or modified test files. |
| {color:green}+1{color} | javac | 8m 4s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 18s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 24s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 24s | The applied patch generated 2
new checkstyle issues (total was 547, now 547). |
| {color:red}-1{color} | whitespace | 0m 2s | The patch has 1 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 31s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:red}-1{color} | findbugs | 2m 40s | The patch appears to introduce 1
new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 25s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 213m 33s | Tests failed in hadoop-hdfs. |
| | | 259m 53s | |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs |
| Failed unit tests | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS |
| | hadoop.hdfs.server.balancer.TestBalancerWithEncryptedTransfer |
| | hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup |
| | hadoop.hdfs.server.blockmanagement.TestBlockManager |
| | hadoop.hdfs.TestGetBlocks |
| | hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes |
| | hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer |
| | hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade |
| | hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes |
| | hadoop.hdfs.server.datanode.TestTriggerBlockReport |
| | hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes |
| | hadoop.hdfs.server.namenode.TestFileTruncate |
| Timed out tests | org.apache.hadoop.hdfs.server.balancer.TestBalancer |
| | org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12752629/HDFS-8967.000.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 4cbbfa2 |
| Pre-patch Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/12150/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/12150/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-HDFS-Build/12150/artifact/patchprocess/whitespace.txt
|
| Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/12150/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12150/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/12150/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12150/console |
This message was automatically generated.
> Create a BlockManagerLock class to represent the lock used in the BlockManager
> --
>
> Key: HDFS-8967
> URL: https://issues.apache.org/jira/browse/HDFS-8967
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Attachments: HDFS-8967.000.patch
>
>
> This jira proposes to create a {{BlockManagerLock}} class to represent the
> lock used in {{BlockManager}}.
> Currently it directly points to the {{FSNamesystem}} lock thus there are no
> functionality changes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8777) Erasure Coding: add tests for taking snapshots on EC files
[ https://issues.apache.org/jira/browse/HDFS-8777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14716131#comment-14716131 ] Rakesh R commented on HDFS-8777: Thanks [~zhz] for the interest. I will rebase the patch once HDFS-8833 is finalized. > Erasure Coding: add tests for taking snapshots on EC files > -- > > Key: HDFS-8777 > URL: https://issues.apache.org/jira/browse/HDFS-8777 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Jing Zhao >Assignee: Rakesh R > Labels: test > Attachments: HDFS-8777-HDFS-7285-00.patch > > > We need to add more tests for (EC + snapshots). The tests need to verify the > fsimage saving/loading is correct. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-8972) EINVAL Invalid argument when RAM_DISK usage 90%+
Xu Chen created HDFS-8972:
-
Summary: EINVAL Invalid argument when RAM_DISK usage 90%+
Key: HDFS-8972
URL: https://issues.apache.org/jira/browse/HDFS-8972
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Xu Chen
Priority: Critical
the directory which is use LAZY_PERSIST policy , so use "df" command look up
tmpfs is usage >=90% , run spark,hive or mapreduce application , Datanode come
out following exception
{code}
2015-08-26 17:37:34,123 WARN org.apache.hadoop.io.ReadaheadPool: Failed
readahead on null
EINVAL: Invalid argument
at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native
Method)
at
org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
at
org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
at
org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
{code}
And the application is slowly than without exception 25%
Regards
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8971) Remove guards when calling LOG.debug() and LOG.trace() in shortcircuit package
[
https://issues.apache.org/jira/browse/HDFS-8971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HDFS-8971:
Target Version/s: 2.8.0
Fix Version/s: (was: 2.8.0)
> Remove guards when calling LOG.debug() and LOG.trace() in shortcircuit package
> --
>
> Key: HDFS-8971
> URL: https://issues.apache.org/jira/browse/HDFS-8971
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: build
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
>
> We moved the {{shortcircuit}} package from {{hadoop-hdfs}} to
> {{hadoop-hdfs-client}} module in JIRA
> [HDFS-8934|https://issues.apache.org/jira/browse/HDFS-8934] and
> [HDFS-8951|https://issues.apache.org/jira/browse/HDFS-8951]. Meanwhile, we
> also replaced the _log4j_ log with _slf4j_ logger. There were existing code
> in the {{shortcircuit}} package to guard the log when calling {{LOG.debug()}}
> and {{LOG.trace()}}, e.g. in {{ShortCircuitCache.java}}, we have code like
> this:
> {code}
> 724if (LOG.isTraceEnabled()) {
> 725 LOG.trace(this + ": found waitable for " + key);
> 726}
> {code}
> In _slf4j_, this kind of guard is not necessary. We should clean the code by
> removing the guard.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8970) Clean up checkstyle issues in HDFS-8934
[
https://issues.apache.org/jira/browse/HDFS-8970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HDFS-8970:
Fix Version/s: (was: 2.8.0)
> Clean up checkstyle issues in HDFS-8934
> ---
>
> Key: HDFS-8970
> URL: https://issues.apache.org/jira/browse/HDFS-8970
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: build
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
>
> We moved the {{ShortCircuitShm}} class into the {{hdfs-client}} module in
> HDFS-8934. There are some checkstyle issues which are not fixed in that jira
> since it only tracked the effort of moving. We need to fix the checkstyle
> issues in {{ShortCircuitShm}} related classes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8970) Clean up checkstyle issues in HDFS-8934
[
https://issues.apache.org/jira/browse/HDFS-8970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HDFS-8970:
Target Version/s: 2.8.0
> Clean up checkstyle issues in HDFS-8934
> ---
>
> Key: HDFS-8970
> URL: https://issues.apache.org/jira/browse/HDFS-8970
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: build
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
>
> We moved the {{ShortCircuitShm}} class into the {{hdfs-client}} module in
> HDFS-8934. There are some checkstyle issues which are not fixed in that jira
> since it only tracked the effort of moving. We need to fix the checkstyle
> issues in {{ShortCircuitShm}} related classes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8971) Remove guards when calling LOG.debug() and LOG.trace() in slf4j
[
https://issues.apache.org/jira/browse/HDFS-8971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HDFS-8971:
Description:
We moved the {{shortcircuit}} package from {{hadoop-hdfs}} to
{{hadoop-hdfs-client}} module in JIRA
[HDFS-8934|https://issues.apache.org/jira/browse/HDFS-8934] and
[HDFS-8951|https://issues.apache.org/jira/browse/HDFS-8951]. Meanwhile, we also
replaced the _log4j_ log with _slf4j_ logger. There were existing code in the
{{shortcircuit}} package to guard the log when calling {{LOG.debug()}} and
{{LOG.trace()}}, e.g. in {{ShortCircuitCache.java}}, we have code like this:
{code}
724if (LOG.isTraceEnabled()) {
725 LOG.trace(this + ": found waitable for " + key);
726}
{code}
In _slf4j_, this kind of guard is not necessary. We should clean the code by
removing the guard.
was:This jira tracks the effort of moving the {{ShortCircuitShm}} class into
the hdfs-client module.
> Remove guards when calling LOG.debug() and LOG.trace() in slf4j
> ---
>
> Key: HDFS-8971
> URL: https://issues.apache.org/jira/browse/HDFS-8971
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: build
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Fix For: 2.8.0
>
>
> We moved the {{shortcircuit}} package from {{hadoop-hdfs}} to
> {{hadoop-hdfs-client}} module in JIRA
> [HDFS-8934|https://issues.apache.org/jira/browse/HDFS-8934] and
> [HDFS-8951|https://issues.apache.org/jira/browse/HDFS-8951]. Meanwhile, we
> also replaced the _log4j_ log with _slf4j_ logger. There were existing code
> in the {{shortcircuit}} package to guard the log when calling {{LOG.debug()}}
> and {{LOG.trace()}}, e.g. in {{ShortCircuitCache.java}}, we have code like
> this:
> {code}
> 724if (LOG.isTraceEnabled()) {
> 725 LOG.trace(this + ": found waitable for " + key);
> 726}
> {code}
> In _slf4j_, this kind of guard is not necessary. We should clean the code by
> removing the guard.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8971) Remove guards when calling LOG.debug() and LOG.trace() in shortcircuit package
[
https://issues.apache.org/jira/browse/HDFS-8971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HDFS-8971:
Summary: Remove guards when calling LOG.debug() and LOG.trace() in
shortcircuit package (was: Remove guards when calling LOG.debug() and
LOG.trace() in slf4j)
> Remove guards when calling LOG.debug() and LOG.trace() in shortcircuit package
> --
>
> Key: HDFS-8971
> URL: https://issues.apache.org/jira/browse/HDFS-8971
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: build
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Fix For: 2.8.0
>
>
> We moved the {{shortcircuit}} package from {{hadoop-hdfs}} to
> {{hadoop-hdfs-client}} module in JIRA
> [HDFS-8934|https://issues.apache.org/jira/browse/HDFS-8934] and
> [HDFS-8951|https://issues.apache.org/jira/browse/HDFS-8951]. Meanwhile, we
> also replaced the _log4j_ log with _slf4j_ logger. There were existing code
> in the {{shortcircuit}} package to guard the log when calling {{LOG.debug()}}
> and {{LOG.trace()}}, e.g. in {{ShortCircuitCache.java}}, we have code like
> this:
> {code}
> 724if (LOG.isTraceEnabled()) {
> 725 LOG.trace(this + ": found waitable for " + key);
> 726}
> {code}
> In _slf4j_, this kind of guard is not necessary. We should clean the code by
> removing the guard.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8850) VolumeScanner thread exits with exception if there is no block pool to be scanned but there are suspicious blocks
[ https://issues.apache.org/jira/browse/HDFS-8850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14716118#comment-14716118 ] Xu Chen commented on HDFS-8850: --- +1 LGTM > VolumeScanner thread exits with exception if there is no block pool to be > scanned but there are suspicious blocks > - > > Key: HDFS-8850 > URL: https://issues.apache.org/jira/browse/HDFS-8850 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.7.0 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Fix For: 2.8.0 > > Attachments: HDFS-8850.001.patch > > > The VolumeScanner threads inside the BlockScanner exit with an exception if > there is no block pool to be scanned but there are suspicious blocks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8945) Update the description about replica placement in HDFS Architecture documentation
[
https://issues.apache.org/jira/browse/HDFS-8945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14716115#comment-14716115
]
Hadoop QA commented on HDFS-8945:
-
\\
\\
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 3m 2s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | release audit | 0m 21s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | site | 2m 56s | Site still builds. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| | | 6m 23s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12752660/HDFS-8945.002.patch |
| Optional Tests | site |
| git revision | trunk / 4cbbfa2 |
| Java | 1.7.0_55 |
| uname | Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12155/console |
This message was automatically generated.
> Update the description about replica placement in HDFS Architecture
> documentation
> -
>
> Key: HDFS-8945
> URL: https://issues.apache.org/jira/browse/HDFS-8945
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: documentation
>Reporter: Masatake Iwasaki
>Assignee: Masatake Iwasaki
>Priority: Minor
> Attachments: HDFS-8945.001.patch, HDFS-8945.002.patch
>
>
> The description about replica placement should have
> * Explanation about storage types and storage policies should be added
> * placement policy for replication factor greater than 4
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-8971) Remove guards when calling LOG.debug() and LOG.trace() in slf4j
Mingliang Liu created HDFS-8971:
---
Summary: Remove guards when calling LOG.debug() and LOG.trace() in
slf4j
Key: HDFS-8971
URL: https://issues.apache.org/jira/browse/HDFS-8971
Project: Hadoop HDFS
Issue Type: Sub-task
Components: build
Reporter: Mingliang Liu
Assignee: Mingliang Liu
Fix For: 2.8.0
This jira tracks the effort of moving the {{ShortCircuitShm}} class into the
hdfs-client module.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8970) Clean up checkstyle issues in HDFS-8934
[
https://issues.apache.org/jira/browse/HDFS-8970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HDFS-8970:
Description: We moved the {{ShortCircuitShm}} class into the
{{hdfs-client}} module in HDFS-8934. There are some checkstyle issues which are
not fixed in that jira since it only tracked the effort of moving. We need to
fix the checkstyle issues in {{ShortCircuitShm}} related classes. (was: This
jira tracks the effort of moving the {{ShortCircuitShm}} class into the
hdfs-client module.)
> Clean up checkstyle issues in HDFS-8934
> ---
>
> Key: HDFS-8970
> URL: https://issues.apache.org/jira/browse/HDFS-8970
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: build
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Fix For: 2.8.0
>
>
> We moved the {{ShortCircuitShm}} class into the {{hdfs-client}} module in
> HDFS-8934. There are some checkstyle issues which are not fixed in that jira
> since it only tracked the effort of moving. We need to fix the checkstyle
> issues in {{ShortCircuitShm}} related classes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-8970) Clean up checkstyle issues in HDFS-8934
Mingliang Liu created HDFS-8970:
---
Summary: Clean up checkstyle issues in HDFS-8934
Key: HDFS-8970
URL: https://issues.apache.org/jira/browse/HDFS-8970
Project: Hadoop HDFS
Issue Type: Sub-task
Components: build
Reporter: Mingliang Liu
Assignee: Mingliang Liu
Fix For: 2.8.0
This jira tracks the effort of moving the {{ShortCircuitShm}} class into the
hdfs-client module.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8945) Update the description about replica placement in HDFS Architecture documentation
[ https://issues.apache.org/jira/browse/HDFS-8945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14716108#comment-14716108 ] Masatake Iwasaki commented on HDFS-8945: I attached 002. I did not add explanation about BlockPlacementPolicy because it is private API. > Update the description about replica placement in HDFS Architecture > documentation > - > > Key: HDFS-8945 > URL: https://issues.apache.org/jira/browse/HDFS-8945 > Project: Hadoop HDFS > Issue Type: Improvement > Components: documentation >Reporter: Masatake Iwasaki >Assignee: Masatake Iwasaki >Priority: Minor > Attachments: HDFS-8945.001.patch, HDFS-8945.002.patch > > > The description about replica placement should have > * Explanation about storage types and storage policies should be added > * placement policy for replication factor greater than 4 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8829) DataNode sets SO_RCVBUF explicitly is disabling tcp auto-tuning
[
https://issues.apache.org/jira/browse/HDFS-8829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
He Tianyi updated HDFS-8829:
Attachment: HDFS-8829.0003.patch
Yes, right.
Thanks for comments.
I've updated the patch.
> DataNode sets SO_RCVBUF explicitly is disabling tcp auto-tuning
> ---
>
> Key: HDFS-8829
> URL: https://issues.apache.org/jira/browse/HDFS-8829
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 2.3.0, 2.6.0
>Reporter: He Tianyi
>Assignee: He Tianyi
> Attachments: HDFS-8829.0001.patch, HDFS-8829.0002.patch,
> HDFS-8829.0003.patch
>
>
> {code:java}
> private void initDataXceiver(Configuration conf) throws IOException {
> // find free port or use privileged port provided
> TcpPeerServer tcpPeerServer;
> if (secureResources != null) {
> tcpPeerServer = new TcpPeerServer(secureResources);
> } else {
> tcpPeerServer = new TcpPeerServer(dnConf.socketWriteTimeout,
> DataNode.getStreamingAddr(conf));
> }
>
> tcpPeerServer.setReceiveBufferSize(HdfsConstants.DEFAULT_DATA_SOCKET_SIZE);
> {code}
> The last line sets SO_RCVBUF explicitly, thus disabling tcp auto-tuning on
> some system.
> Shall we make this behavior configurable?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8945) Update the description about replica placement in HDFS Architecture documentation
[ https://issues.apache.org/jira/browse/HDFS-8945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated HDFS-8945: --- Attachment: HDFS-8945.002.patch > Update the description about replica placement in HDFS Architecture > documentation > - > > Key: HDFS-8945 > URL: https://issues.apache.org/jira/browse/HDFS-8945 > Project: Hadoop HDFS > Issue Type: Improvement > Components: documentation >Reporter: Masatake Iwasaki >Assignee: Masatake Iwasaki >Priority: Minor > Attachments: HDFS-8945.001.patch, HDFS-8945.002.patch > > > The description about replica placement should have > * Explanation about storage types and storage policies should be added > * placement policy for replication factor greater than 4 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8925) Move BlockReader to hdfs-client
[
https://issues.apache.org/jira/browse/HDFS-8925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HDFS-8925:
Description: This jira tracks the effort of moving the {{BlockReader}}
class into the hdfs-client module. We also move {{BlockReaderLocal}} class
which implements the {{BlockReader}} interface to {{hdfs-client}} module.
(was: This jira tracks the effort of moving the {{BlockReader}} class into the
hdfs-client module.)
> Move BlockReader to hdfs-client
> ---
>
> Key: HDFS-8925
> URL: https://issues.apache.org/jira/browse/HDFS-8925
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: build
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HDFS-8925.000.patch
>
>
> This jira tracks the effort of moving the {{BlockReader}} class into the
> hdfs-client module. We also move {{BlockReaderLocal}} class which implements
> the {{BlockReader}} interface to {{hdfs-client}} module.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8965) Harden edit log reading code against out of memory errors
[
https://issues.apache.org/jira/browse/HDFS-8965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14716101#comment-14716101
]
Hadoop QA commented on HDFS-8965:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 18m 27s | Pre-patch trunk has 4 extant
Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 52s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 59s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 24s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 36s | The applied patch generated
12 new checkstyle issues (total was 385, now 392). |
| {color:green}+1{color} | whitespace | 0m 1s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 30s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:red}-1{color} | findbugs | 3m 25s | The patch appears to introduce 1
new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 7s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 160m 49s | Tests failed in hadoop-hdfs. |
| {color:red}-1{color} | hdfs tests | 0m 14s | Tests failed in bkjournal. |
| | | 208m 8s | |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs |
| Failed unit tests | hadoop.hdfs.TestRollingUpgrade |
| | hadoop.hdfs.crypto.TestHdfsCryptoStreams |
| | hadoop.hdfs.server.namenode.TestFSNamesystem |
| | hadoop.hdfs.qjournal.server.TestJournal |
| Failed build | bkjournal |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12752628/HDFS-8965.003.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 4cbbfa2 |
| Pre-patch Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/12151/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/12151/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
|
| Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/12151/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12151/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| bkjournal test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12151/artifact/patchprocess/testrun_bkjournal.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/12151/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12151/console |
This message was automatically generated.
> Harden edit log reading code against out of memory errors
> -
>
> Key: HDFS-8965
> URL: https://issues.apache.org/jira/browse/HDFS-8965
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 2.0.0-alpha
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Attachments: HDFS-8965.001.patch, HDFS-8965.002.patch,
> HDFS-8965.003.patch
>
>
> We should harden the edit log reading code against out of memory errors. Now
> that each op has a length prefix and a checksum, we can validate the checksum
> before trying to load the Op data. This should avoid out of memory errors
> when trying to load garbage data as Op data.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8938) Refactor BlockManager in blockmanagement
[
https://issues.apache.org/jira/browse/HDFS-8938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HDFS-8938:
Description:
This jira tracks the effort of refactoring inner classes
{{BlockManager$BlockToMarkCorrupt}} and {{BlockManager$ReplicationWork}} in
{{hdfs.server.blockmanagement}} package. As the line number of {{BlockManager}}
is getting larger than 2000, we can move those two inner classes out of the it.
Meanwhile, the logic in method {{computeReplicationWorkForBlocks}} can be
simplified if we extract code sections to _schedule replication_ and to
_validate replication work_ to private helper methods respectively.
was:This lira tracks the effort of refactoring the {{BlockManager}} in
{{hdfs.server.blockmanagement}} package.
> Refactor BlockManager in blockmanagement
>
>
> Key: HDFS-8938
> URL: https://issues.apache.org/jira/browse/HDFS-8938
> Project: Hadoop HDFS
> Issue Type: Task
> Components: build
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HDFS-8938.000.patch, HDFS-8938.001.patch,
> HDFS-8938.002.patch, HDFS-8938.003.patch, HDFS-8938.004.patch,
> HDFS-8938.005.patch
>
>
> This jira tracks the effort of refactoring inner classes
> {{BlockManager$BlockToMarkCorrupt}} and {{BlockManager$ReplicationWork}} in
> {{hdfs.server.blockmanagement}} package. As the line number of
> {{BlockManager}} is getting larger than 2000, we can move those two inner
> classes out of the it.
> Meanwhile, the logic in method {{computeReplicationWorkForBlocks}} can be
> simplified if we extract code sections to _schedule replication_ and to
> _validate replication work_ to private helper methods respectively.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8938) Refactor BlockManager in blockmanagement
[
https://issues.apache.org/jira/browse/HDFS-8938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HDFS-8938:
Description:
This jira tracks the effort of refactoring inner classes
{{BlockManager$BlockToMarkCorrupt}} and {{BlockManager$ReplicationWork}} in
{{hdfs.server.blockmanagement}} package. As the line number of {{BlockManager}}
is getting larger than 2000, we can move those two inner classes out of the it.
Meanwhile, the logic in method {{computeReplicationWorkForBlocks}} can be
simplified if we extract code sections to _schedule replication_ and to
_validate replication work_ to private helper methods respectively.
was:
This jira tracks the effort of refactoring inner classes
{{BlockManager$BlockToMarkCorrupt}} and {{BlockManager$ReplicationWork}} in
{{hdfs.server.blockmanagement}} package. As the line number of {{BlockManager}}
is getting larger than 2000, we can move those two inner classes out of the it.
Meanwhile, the logic in method {{computeReplicationWorkForBlocks}} can be
simplified if we extract code sections to _schedule replication_ and to
_validate replication work_ to private helper methods respectively.
> Refactor BlockManager in blockmanagement
>
>
> Key: HDFS-8938
> URL: https://issues.apache.org/jira/browse/HDFS-8938
> Project: Hadoop HDFS
> Issue Type: Task
> Components: build
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HDFS-8938.000.patch, HDFS-8938.001.patch,
> HDFS-8938.002.patch, HDFS-8938.003.patch, HDFS-8938.004.patch,
> HDFS-8938.005.patch
>
>
> This jira tracks the effort of refactoring inner classes
> {{BlockManager$BlockToMarkCorrupt}} and {{BlockManager$ReplicationWork}} in
> {{hdfs.server.blockmanagement}} package. As the line number of
> {{BlockManager}} is getting larger than 2000, we can move those two inner
> classes out of the it.
> Meanwhile, the logic in method {{computeReplicationWorkForBlocks}} can be
> simplified if we extract code sections to _schedule replication_ and to
> _validate replication work_ to private helper methods respectively.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8962) Fix checkstyle and whitespace issues in HDFS-8803
[
https://issues.apache.org/jira/browse/HDFS-8962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HDFS-8962:
Attachment: HDFS-8962.002.patch
> Fix checkstyle and whitespace issues in HDFS-8803
> -
>
> Key: HDFS-8962
> URL: https://issues.apache.org/jira/browse/HDFS-8962
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: build
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HDFS-8962.000.patch, HDFS-8962.001.patch,
> HDFS-8962.002.patch
>
>
> This jira tracks the effort of fixing whitespace and checkstyle issues after
> moving {{DfsClientConf}} to the hdfs-client module.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8248) Store INodeId instead of the INodeFile object in BlockInfoContiguous
[
https://issues.apache.org/jira/browse/HDFS-8248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14716092#comment-14716092
]
Hudson commented on HDFS-8248:
--
SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #308 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/308/])
HDFS-8248. Store INodeId instead of the INodeFile object in
BlockInfoContiguous. Contributed by Haohui Mai. (wheat9: rev
4cbbfa2220e884e91bf18ad1cc2f3b11f895f8c9)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlocksMap.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCommitBlockSynchronization.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/Namesystem.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfo.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeId.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestSnapshotDeletion.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockCollection.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockInfo.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestSnapshotBlocksMap.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DecommissionManager.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/SequentialBlockIdGenerator.java
> Store INodeId instead of the INodeFile object in BlockInfoContiguous
>
>
> Key: HDFS-8248
> URL: https://issues.apache.org/jira/browse/HDFS-8248
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Fix For: 2.8.0
>
> Attachments: HDFS-8248.000.patch, HDFS-8248.001.patch,
> HDFS-8248.002.patch, HDFS-8248.003.patch, HDFS-8248.004.patch,
> HDFS-8248.005.patch, HDFS-8248.006.patch, HDFS-8248.007.patch,
> HDFS-8248.008.patch
>
>
> Currently the namespace and the block manager are tightly coupled together.
> There are two couplings in terms of implementation:
> 1. The {{BlockInfoContiguous}} stores a reference of the {{INodeFile}} that
> owns the block, so that the block manager can look up the corresponding file
> when replicating blocks, recovering from pipeline failures, etc.
> 1. The {{INodeFile}} stores {{BlockInfoContiguous}} objects that the file
> owns.
> Decoupling the namespace and the block manager allows the BM to be separated
> out from the Java heap or even as a standalone process. This jira proposes to
> remove the first coupling by storing the id of the inode instead of the
> object reference of {{INodeFile}} in the {{BlockInfoContiguous}} class.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8909) Erasure coding: update BlockInfoContiguousUC and BlockInfoStripedUC to use BlockUnderConstructionFeature
[
https://issues.apache.org/jira/browse/HDFS-8909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14716089#comment-14716089
]
Hadoop QA commented on HDFS-8909:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 15m 26s | Findbugs (version ) appears to
be broken on HDFS-7285. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 12 new or modified test files. |
| {color:green}+1{color} | javac | 7m 45s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 57s | There were no new javadoc
warning messages. |
| {color:red}-1{color} | release audit | 0m 15s | The applied patch generated
1 release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 32s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 13s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 31s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:red}-1{color} | findbugs | 2m 43s | The patch appears to introduce 4
new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 9s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 183m 35s | Tests failed in hadoop-hdfs. |
| | | 225m 48s | |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs |
| Failed unit tests | hadoop.hdfs.TestSafeModeWithStripedFile |
| | hadoop.hdfs.TestWriteStripedFileWithFailure |
| | hadoop.hdfs.TestRollingUpgrade |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12752620/HDFS-8909-HDFS-7285.004.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | HDFS-7285 / 067ec8c |
| Release Audit |
https://builds.apache.org/job/PreCommit-HDFS-Build/12149/artifact/patchprocess/patchReleaseAuditProblems.txt
|
| Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/12149/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12149/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/12149/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12149/console |
This message was automatically generated.
> Erasure coding: update BlockInfoContiguousUC and BlockInfoStripedUC to use
> BlockUnderConstructionFeature
>
>
> Key: HDFS-8909
> URL: https://issues.apache.org/jira/browse/HDFS-8909
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: HDFS-7285
>Reporter: Zhe Zhang
>Assignee: Jing Zhao
> Attachments: HDFS-8909-HDFS-7285.003.patch,
> HDFS-8909-HDFS-7285.004.patch, HDFS-8909.000.patch, HDFS-8909.001.patch,
> HDFS-8909.002.patch, HDFS-8909.003.patch
>
>
> HDFS-8801 converts {{BlockInfoUC}} as a feature. We should consolidate
> {{BlockInfoContiguousUC}} and {{BlockInfoStripedUC}} logics to use this
> feature.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8932) NPE thrown in NameNode when try to get "TotalSyncCount" metric before editLogStream initialization
[ https://issues.apache.org/jira/browse/HDFS-8932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14716087#comment-14716087 ] Anu Engineer commented on HDFS-8932: thx for pointing it out, I just filed HDFS-8969 > NPE thrown in NameNode when try to get "TotalSyncCount" metric before > editLogStream initialization > -- > > Key: HDFS-8932 > URL: https://issues.apache.org/jira/browse/HDFS-8932 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Surendra Singh Lilhore >Assignee: Surendra Singh Lilhore > Fix For: 2.8.0 > > Attachments: HDFS-8932.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-8969) Inconsistent synchronization of FSEditLog.editLogStream;
Anu Engineer created HDFS-8969: -- Summary: Inconsistent synchronization of FSEditLog.editLogStream; Key: HDFS-8969 URL: https://issues.apache.org/jira/browse/HDFS-8969 Project: Hadoop HDFS Issue Type: Bug Components: HDFS Affects Versions: 2.8.0 Reporter: Anu Engineer Assignee: Anu Engineer Fix Findbug warning on Synchronization. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8962) Fix checkstyle and whitespace issues in HDFS-8803
[
https://issues.apache.org/jira/browse/HDFS-8962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HDFS-8962:
Attachment: HDFS-8962.001.patch
> Fix checkstyle and whitespace issues in HDFS-8803
> -
>
> Key: HDFS-8962
> URL: https://issues.apache.org/jira/browse/HDFS-8962
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: build
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HDFS-8962.000.patch, HDFS-8962.001.patch
>
>
> This jira tracks the effort of fixing whitespace and checkstyle issues after
> moving {{DfsClientConf}} to the hdfs-client module.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8932) NPE thrown in NameNode when try to get "TotalSyncCount" metric before editLogStream initialization
[ https://issues.apache.org/jira/browse/HDFS-8932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14716079#comment-14716079 ] Brahma Reddy Battula commented on HDFS-8932: It appears that this jira introduces one new findbugs warnings in Jenkins. https://builds.apache.org/job/PreCommit-HDFS-Build/12138/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html > NPE thrown in NameNode when try to get "TotalSyncCount" metric before > editLogStream initialization > -- > > Key: HDFS-8932 > URL: https://issues.apache.org/jira/browse/HDFS-8932 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Surendra Singh Lilhore >Assignee: Surendra Singh Lilhore > Fix For: 2.8.0 > > Attachments: HDFS-8932.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8929) Add a metric to expose the timestamp of the last journal
[
https://issues.apache.org/jira/browse/HDFS-8929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14716071#comment-14716071
]
Brahma Reddy Battula commented on HDFS-8929:
[~surendrasingh] thanks for updating patch..
Again you missed to correct *transections* in following two lines., it should
be *transactions*
1) `LastJournalTimestamp` | The timestamp of last *successfully written
transections*
2) @Metric("The timestamp of last *successfully written transections* ")
and 3) please remove the white space also..
> Add a metric to expose the timestamp of the last journal
>
>
> Key: HDFS-8929
> URL: https://issues.apache.org/jira/browse/HDFS-8929
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: journal-node
>Reporter: Akira AJISAKA
>Assignee: Surendra Singh Lilhore
> Attachments: HDFS-8929-001.patch, HDFS-8929-002.patch
>
>
> If there are three JNs and only one JN is failing to journal, we can detect
> it by monitoring the difference of the last written transaction id among JNs
> from NN WebUI or JN metrics. However, it's difficult to define the threshold
> to alert because the increase rate of the number of transaction depends on
> how busy the cluster is. Therefore I'd like to propose a metric to expose the
> timestamp of the last journal. That way we can easily alert if a JN is
> failing to journal for some fixed period.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8896) DataNode object isn't GCed when shutdown, because it has GC root in ShutdownHookManager
[
https://issues.apache.org/jira/browse/HDFS-8896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14716061#comment-14716061
]
Hudson commented on HDFS-8896:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #299 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/299/])
HDFS-8896. DataNode object isn't GCed when shutdown, because it has GC root in
ShutdownHookManager. Contributed by Walter Su. (jing9: rev
f44b599003bb79b1ec9b92e7546546523ec01676)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java
*
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ShutdownHookManager.java
> DataNode object isn't GCed when shutdown, because it has GC root in
> ShutdownHookManager
> ---
>
> Key: HDFS-8896
> URL: https://issues.apache.org/jira/browse/HDFS-8896
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Walter Su
>Assignee: Walter Su
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HDFS-8896.01.patch, screenshot_1.PNG, screenshot_2.PNG
>
>
> The anonymous {{Thread}} object created in {{ShutdownHookManager}} is a GC
> root.
> screenshot_1 shows how DN object be traced to the GC root.
> It's not a problem in production.
> It's a problem in test, especially when MiniDFSCluster starts/shutdowns many
> DNs, which could cause {{OutOfMemoryError}}.
> screenshot_2 shows many DN objects are not GCed when run the test of
> HDFS-8838.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8248) Store INodeId instead of the INodeFile object in BlockInfoContiguous
[
https://issues.apache.org/jira/browse/HDFS-8248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14716058#comment-14716058
]
Hudson commented on HDFS-8248:
--
SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2257 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2257/])
HDFS-8248. Store INodeId instead of the INodeFile object in
BlockInfoContiguous. Contributed by Haohui Mai. (wheat9: rev
4cbbfa2220e884e91bf18ad1cc2f3b11f895f8c9)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/Namesystem.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/SequentialBlockIdGenerator.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestSnapshotDeletion.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlocksMap.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DecommissionManager.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockInfo.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeId.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestSnapshotBlocksMap.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCommitBlockSynchronization.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockCollection.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfo.java
> Store INodeId instead of the INodeFile object in BlockInfoContiguous
>
>
> Key: HDFS-8248
> URL: https://issues.apache.org/jira/browse/HDFS-8248
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Fix For: 2.8.0
>
> Attachments: HDFS-8248.000.patch, HDFS-8248.001.patch,
> HDFS-8248.002.patch, HDFS-8248.003.patch, HDFS-8248.004.patch,
> HDFS-8248.005.patch, HDFS-8248.006.patch, HDFS-8248.007.patch,
> HDFS-8248.008.patch
>
>
> Currently the namespace and the block manager are tightly coupled together.
> There are two couplings in terms of implementation:
> 1. The {{BlockInfoContiguous}} stores a reference of the {{INodeFile}} that
> owns the block, so that the block manager can look up the corresponding file
> when replicating blocks, recovering from pipeline failures, etc.
> 1. The {{INodeFile}} stores {{BlockInfoContiguous}} objects that the file
> owns.
> Decoupling the namespace and the block manager allows the BM to be separated
> out from the Java heap or even as a standalone process. This jira proposes to
> remove the first coupling by storing the id of the inode instead of the
> object reference of {{INodeFile}} in the {{BlockInfoContiguous}} class.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8896) DataNode object isn't GCed when shutdown, because it has GC root in ShutdownHookManager
[
https://issues.apache.org/jira/browse/HDFS-8896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14716041#comment-14716041
]
Hudson commented on HDFS-8896:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk #2237 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/2237/])
HDFS-8896. DataNode object isn't GCed when shutdown, because it has GC root in
ShutdownHookManager. Contributed by Walter Su. (jing9: rev
f44b599003bb79b1ec9b92e7546546523ec01676)
*
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ShutdownHookManager.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java
> DataNode object isn't GCed when shutdown, because it has GC root in
> ShutdownHookManager
> ---
>
> Key: HDFS-8896
> URL: https://issues.apache.org/jira/browse/HDFS-8896
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Walter Su
>Assignee: Walter Su
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HDFS-8896.01.patch, screenshot_1.PNG, screenshot_2.PNG
>
>
> The anonymous {{Thread}} object created in {{ShutdownHookManager}} is a GC
> root.
> screenshot_1 shows how DN object be traced to the GC root.
> It's not a problem in production.
> It's a problem in test, especially when MiniDFSCluster starts/shutdowns many
> DNs, which could cause {{OutOfMemoryError}}.
> screenshot_2 shows many DN objects are not GCed when run the test of
> HDFS-8838.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8833) Erasure coding: store EC schema and cell size in INodeFile and eliminate notion of EC zones
[
https://issues.apache.org/jira/browse/HDFS-8833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14716015#comment-14716015
]
Kai Zheng commented on HDFS-8833:
-
Thanks for the great discussions here and I may need some time to understand it
fully.
Looks like it's hard to get an perfect approach out, now we may have one but
the required change is large. I'm wondering if we could move on in this aspect
in follow-on phase? Some experience from our side, the current codes in
HDFS-7285 branch works great very reliably and stably under extensive perf test
loads. I don't have insights here about which way is better, just thought it
would be great if we could move on. Have a solution in hand first and get the
whole feature exercised as earlier as possible by experimental deploying, we
may have more inputs to evolve. Apologize if anything bad. Thanks.
> Erasure coding: store EC schema and cell size in INodeFile and eliminate
> notion of EC zones
> ---
>
> Key: HDFS-8833
> URL: https://issues.apache.org/jira/browse/HDFS-8833
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: HDFS-7285
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Attachments: HDFS-8833-HDFS-7285-merge.00.patch,
> HDFS-8833-HDFS-7285-merge.01.patch, HDFS-8833-HDFS-7285.02.patch
>
>
> We have [discussed |
> https://issues.apache.org/jira/browse/HDFS-7285?focusedCommentId=14357754&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14357754]
> storing EC schema with files instead of EC zones and recently revisited the
> discussion under HDFS-8059.
> As a recap, the _zone_ concept has severe limitations including renaming and
> nested configuration. Those limitations are valid in encryption for security
> reasons and it doesn't make sense to carry them over in EC.
> This JIRA aims to store EC schema and cell size on {{INodeFile}} level. For
> simplicity, we should first implement it as an xattr and consider memory
> optimizations (such as moving it to file header) as a follow-on. We should
> also disable changing EC policy on a non-empty file / dir in the first phase.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8248) Store INodeId instead of the INodeFile object in BlockInfoContiguous
[
https://issues.apache.org/jira/browse/HDFS-8248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14716013#comment-14716013
]
Hudson commented on HDFS-8248:
--
SUCCESS: Integrated in Hadoop-Yarn-trunk #1041 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/1041/])
HDFS-8248. Store INodeId instead of the INodeFile object in
BlockInfoContiguous. Contributed by Haohui Mai. (wheat9: rev
4cbbfa2220e884e91bf18ad1cc2f3b11f895f8c9)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockInfo.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestSnapshotDeletion.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfo.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/SequentialBlockIdGenerator.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeId.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCommitBlockSynchronization.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/Namesystem.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestSnapshotBlocksMap.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockCollection.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlocksMap.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DecommissionManager.java
> Store INodeId instead of the INodeFile object in BlockInfoContiguous
>
>
> Key: HDFS-8248
> URL: https://issues.apache.org/jira/browse/HDFS-8248
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Fix For: 2.8.0
>
> Attachments: HDFS-8248.000.patch, HDFS-8248.001.patch,
> HDFS-8248.002.patch, HDFS-8248.003.patch, HDFS-8248.004.patch,
> HDFS-8248.005.patch, HDFS-8248.006.patch, HDFS-8248.007.patch,
> HDFS-8248.008.patch
>
>
> Currently the namespace and the block manager are tightly coupled together.
> There are two couplings in terms of implementation:
> 1. The {{BlockInfoContiguous}} stores a reference of the {{INodeFile}} that
> owns the block, so that the block manager can look up the corresponding file
> when replicating blocks, recovering from pipeline failures, etc.
> 1. The {{INodeFile}} stores {{BlockInfoContiguous}} objects that the file
> owns.
> Decoupling the namespace and the block manager allows the BM to be separated
> out from the Java heap or even as a standalone process. This jira proposes to
> remove the first coupling by storing the id of the inode instead of the
> object reference of {{INodeFile}} in the {{BlockInfoContiguous}} class.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HDFS-8954) Not balance block in RAM_DISK use lazy_persist policy on datanode local copy
[ https://issues.apache.org/jira/browse/HDFS-8954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang resolved HDFS-8954. --- Resolution: Not A Problem This is working as intended, only the local replica goes to RAM_DISK, the others go to DISK. > Not balance block in RAM_DISK use lazy_persist policy on datanode local copy > - > > Key: HDFS-8954 > URL: https://issues.apache.org/jira/browse/HDFS-8954 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.7.1 >Reporter: Xu Chen >Priority: Critical > > I was try to use Heterogeneous Storage feature and I config RAM_DISK volume > for each DN size of 20g , and set one directory policy of lazy_persist at > HDFS , then command copyFromLocal to load a 79g file to the directory HDFS , > it is have 593 block , and only 141 block in RAM_DISK , I use command "df" to > look tmpfs usage it only my execute command DN fill enough the tmpfs , other > DN tmpfs is empty ! > when I use other machine without datanode installed , it is worked fine all > DN tmpfs is fill enough > Regards -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8962) Fix checkstyle and whitespace issues in HDFS-8803
[
https://issues.apache.org/jira/browse/HDFS-8962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14716009#comment-14716009
]
Hadoop QA commented on HDFS-8962:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 19m 21s | Pre-patch trunk has 6 extant
Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 5 new or modified test files. |
| {color:green}+1{color} | javac | 7m 45s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 2s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 25s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 2m 56s | There were no new checkstyle
issues. |
| {color:red}-1{color} | whitespace | 0m 1s | The patch has 2 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 29s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 4m 34s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 18s | Pre-build of native portion |
| {color:green}+1{color} | hdfs tests | 162m 46s | Tests passed in hadoop-hdfs.
|
| {color:green}+1{color} | hdfs tests | 0m 29s | Tests passed in
hadoop-hdfs-client. |
| | | 213m 43s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12752599/HDFS-8962.000.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f44b599 |
| Pre-patch Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/12146/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
|
| Pre-patch Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/12146/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs-client.html
|
| whitespace |
https://builds.apache.org/job/PreCommit-HDFS-Build/12146/artifact/patchprocess/whitespace.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12146/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| hadoop-hdfs-client test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12146/artifact/patchprocess/testrun_hadoop-hdfs-client.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/12146/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12146/console |
This message was automatically generated.
> Fix checkstyle and whitespace issues in HDFS-8803
> -
>
> Key: HDFS-8962
> URL: https://issues.apache.org/jira/browse/HDFS-8962
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: build
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HDFS-8962.000.patch
>
>
> This jira tracks the effort of fixing whitespace and checkstyle issues after
> moving {{DfsClientConf}} to the hdfs-client module.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8965) Harden edit log reading code against out of memory errors
[
https://issues.apache.org/jira/browse/HDFS-8965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14716004#comment-14716004
]
Andrew Wang commented on HDFS-8965:
---
Hi Colin, thanks for working on this. Seems like the idea is to, for
length-prefixed edit logs, to check the checksum before reading the fields.
Sensible and good change, overall looks good. Mostly nit comments:
* A comment mentions a 1-byte txid, I think you meant opid. Having the expected
format in the comment is super helpful too, thanks. Mind replicating it for the
other two readers too?
* Why move the temp array into a member? It'll probably be stack allocated and
thus no GC pressure, 4KB should fit too.
* Great opportunity to fix the LayoutVersion.EDITS_CHESKUM typo
* LengthPrefixedReader#scanOp, shouldn't this be basically readOp but without
the second pass to readFields? Not sure why it doesn't validate the max edit
size or the checksum. The TODO is relevant.
* Can we get a unit test for this situation?
* Is some code sharing is possible among the three readOp's for getting the
FSEditLogOp, this bit?
{noformat}
byte opCodeByte;
try {
opCodeByte = in.readByte();
} catch (EOFException eof) {
// EOF at an opcode boundary is expected.
return null;
}
FSEditLogOpCodes opCode = FSEditLogOpCodes.fromByte(opCodeByte);
if (opCode == OP_INVALID) {
verifyTerminator();
return null;
}
FSEditLogOp op = cache.get(opCode);
if (op == null) {
throw new IOException("Read invalid opcode " + opCode);
}
{noformat}
> Harden edit log reading code against out of memory errors
> -
>
> Key: HDFS-8965
> URL: https://issues.apache.org/jira/browse/HDFS-8965
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 2.0.0-alpha
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Attachments: HDFS-8965.001.patch, HDFS-8965.002.patch,
> HDFS-8965.003.patch
>
>
> We should harden the edit log reading code against out of memory errors. Now
> that each op has a length prefix and a checksum, we can validate the checksum
> before trying to load the Op data. This should avoid out of memory errors
> when trying to load garbage data as Op data.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7285) Erasure Coding Support inside HDFS
[ https://issues.apache.org/jira/browse/HDFS-7285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14715996#comment-14715996 ] Kai Zheng commented on HDFS-7285: - Thanks [~zhz] for the hard taking and I understand we may take some time to make the merging to trunk happen. Before that, I thought it would be good to keep moving. Think about so many follow-on issues listed in HDFS-8031 even for phase I, not to say we have another phase for non-striping erasure coding support. Having some branches for the effort, I'm wondering which branch better to use for the community to continuously work on and commit the follow-on tasks. Any thought? Thanks. > Erasure Coding Support inside HDFS > -- > > Key: HDFS-7285 > URL: https://issues.apache.org/jira/browse/HDFS-7285 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Weihua Jiang >Assignee: Zhe Zhang > Attachments: Compare-consolidated-20150824.diff, > Consolidated-20150707.patch, Consolidated-20150806.patch, > Consolidated-20150810.patch, ECAnalyzer.py, ECParser.py, > HDFS-7285-initial-PoC.patch, HDFS-7285-merge-consolidated-01.patch, > HDFS-7285-merge-consolidated-trunk-01.patch, > HDFS-7285-merge-consolidated.trunk.03.patch, > HDFS-7285-merge-consolidated.trunk.04.patch, > HDFS-EC-Merge-PoC-20150624.patch, HDFS-EC-merge-consolidated-01.patch, > HDFS-bistriped.patch, HDFSErasureCodingDesign-20141028.pdf, > HDFSErasureCodingDesign-20141217.pdf, HDFSErasureCodingDesign-20150204.pdf, > HDFSErasureCodingDesign-20150206.pdf, HDFSErasureCodingPhaseITestPlan.pdf, > HDFSErasureCodingSystemTestPlan-20150824.pdf, fsimage-analysis-20150105.pdf > > > Erasure Coding (EC) can greatly reduce the storage overhead without sacrifice > of data reliability, comparing to the existing HDFS 3-replica approach. For > example, if we use a 10+4 Reed Solomon coding, we can allow loss of 4 blocks, > with storage overhead only being 40%. This makes EC a quite attractive > alternative for big data storage, particularly for cold data. > Facebook had a related open source project called HDFS-RAID. It used to be > one of the contribute packages in HDFS but had been removed since Hadoop 2.0 > for maintain reason. The drawbacks are: 1) it is on top of HDFS and depends > on MapReduce to do encoding and decoding tasks; 2) it can only be used for > cold files that are intended not to be appended anymore; 3) the pure Java EC > coding implementation is extremely slow in practical use. Due to these, it > might not be a good idea to just bring HDFS-RAID back. > We (Intel and Cloudera) are working on a design to build EC into HDFS that > gets rid of any external dependencies, makes it self-contained and > independently maintained. This design lays the EC feature on the storage type > support and considers compatible with existing HDFS features like caching, > snapshot, encryption, high availability and etc. This design will also > support different EC coding schemes, implementations and policies for > different deployment scenarios. By utilizing advanced libraries (e.g. Intel > ISA-L library), an implementation can greatly improve the performance of EC > encoding/decoding and makes the EC solution even more attractive. We will > post the design document soon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7285) Erasure Coding Support inside HDFS
[ https://issues.apache.org/jira/browse/HDFS-7285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14715985#comment-14715985 ] Kai Zheng commented on HDFS-7285: - It's great to have the complete functional system tests for the striping support. Thanks guys! In our side, we're working on end to end performance tests while implementing new erasure coders and optimizing striping codes. Related tasks: * Implementing ISA-L coder (about 25X than the Java coder, HADOOP-11887, HADOOP-11540); * Implementing a new Java RS coder that’s both faster (6X than the existing Java coder) and compatible with ISA-L coder (HADOOP-12041). * Optimizing client striping input/output stream codes (HDFS-8668 and related), consolidating client stateful and positional reads (HDFS-8957 and related); * Performance benchmark test tools (HDFS-8968 and HADOOP-11588). We're running the perf tests while tuning and optimizing the codes, and will provide the report when it's ready. > Erasure Coding Support inside HDFS > -- > > Key: HDFS-7285 > URL: https://issues.apache.org/jira/browse/HDFS-7285 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Weihua Jiang >Assignee: Zhe Zhang > Attachments: Compare-consolidated-20150824.diff, > Consolidated-20150707.patch, Consolidated-20150806.patch, > Consolidated-20150810.patch, ECAnalyzer.py, ECParser.py, > HDFS-7285-initial-PoC.patch, HDFS-7285-merge-consolidated-01.patch, > HDFS-7285-merge-consolidated-trunk-01.patch, > HDFS-7285-merge-consolidated.trunk.03.patch, > HDFS-7285-merge-consolidated.trunk.04.patch, > HDFS-EC-Merge-PoC-20150624.patch, HDFS-EC-merge-consolidated-01.patch, > HDFS-bistriped.patch, HDFSErasureCodingDesign-20141028.pdf, > HDFSErasureCodingDesign-20141217.pdf, HDFSErasureCodingDesign-20150204.pdf, > HDFSErasureCodingDesign-20150206.pdf, HDFSErasureCodingPhaseITestPlan.pdf, > HDFSErasureCodingSystemTestPlan-20150824.pdf, fsimage-analysis-20150105.pdf > > > Erasure Coding (EC) can greatly reduce the storage overhead without sacrifice > of data reliability, comparing to the existing HDFS 3-replica approach. For > example, if we use a 10+4 Reed Solomon coding, we can allow loss of 4 blocks, > with storage overhead only being 40%. This makes EC a quite attractive > alternative for big data storage, particularly for cold data. > Facebook had a related open source project called HDFS-RAID. It used to be > one of the contribute packages in HDFS but had been removed since Hadoop 2.0 > for maintain reason. The drawbacks are: 1) it is on top of HDFS and depends > on MapReduce to do encoding and decoding tasks; 2) it can only be used for > cold files that are intended not to be appended anymore; 3) the pure Java EC > coding implementation is extremely slow in practical use. Due to these, it > might not be a good idea to just bring HDFS-RAID back. > We (Intel and Cloudera) are working on a design to build EC into HDFS that > gets rid of any external dependencies, makes it self-contained and > independently maintained. This design lays the EC feature on the storage type > support and considers compatible with existing HDFS features like caching, > snapshot, encryption, high availability and etc. This design will also > support different EC coding schemes, implementations and policies for > different deployment scenarios. By utilizing advanced libraries (e.g. Intel > ISA-L library), an implementation can greatly improve the performance of EC > encoding/decoding and makes the EC solution even more attractive. We will > post the design document soon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8248) Store INodeId instead of the INodeFile object in BlockInfoContiguous
[
https://issues.apache.org/jira/browse/HDFS-8248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14715982#comment-14715982
]
Hudson commented on HDFS-8248:
--
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #313 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/313/])
HDFS-8248. Store INodeId instead of the INodeFile object in
BlockInfoContiguous. Contributed by Haohui Mai. (wheat9: rev
4cbbfa2220e884e91bf18ad1cc2f3b11f895f8c9)
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/Namesystem.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockInfo.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCommitBlockSynchronization.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfo.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestSnapshotBlocksMap.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/SequentialBlockIdGenerator.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DecommissionManager.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeId.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestSnapshotDeletion.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockCollection.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlocksMap.java
> Store INodeId instead of the INodeFile object in BlockInfoContiguous
>
>
> Key: HDFS-8248
> URL: https://issues.apache.org/jira/browse/HDFS-8248
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Fix For: 2.8.0
>
> Attachments: HDFS-8248.000.patch, HDFS-8248.001.patch,
> HDFS-8248.002.patch, HDFS-8248.003.patch, HDFS-8248.004.patch,
> HDFS-8248.005.patch, HDFS-8248.006.patch, HDFS-8248.007.patch,
> HDFS-8248.008.patch
>
>
> Currently the namespace and the block manager are tightly coupled together.
> There are two couplings in terms of implementation:
> 1. The {{BlockInfoContiguous}} stores a reference of the {{INodeFile}} that
> owns the block, so that the block manager can look up the corresponding file
> when replicating blocks, recovering from pipeline failures, etc.
> 1. The {{INodeFile}} stores {{BlockInfoContiguous}} objects that the file
> owns.
> Decoupling the namespace and the block manager allows the BM to be separated
> out from the Java heap or even as a standalone process. This jira proposes to
> remove the first coupling by storing the id of the inode instead of the
> object reference of {{INodeFile}} in the {{BlockInfoContiguous}} class.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8896) DataNode object isn't GCed when shutdown, because it has GC root in ShutdownHookManager
[
https://issues.apache.org/jira/browse/HDFS-8896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14715980#comment-14715980
]
Hudson commented on HDFS-8896:
--
SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #307 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/307/])
HDFS-8896. DataNode object isn't GCed when shutdown, because it has GC root in
ShutdownHookManager. Contributed by Walter Su. (jing9: rev
f44b599003bb79b1ec9b92e7546546523ec01676)
*
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ShutdownHookManager.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
> DataNode object isn't GCed when shutdown, because it has GC root in
> ShutdownHookManager
> ---
>
> Key: HDFS-8896
> URL: https://issues.apache.org/jira/browse/HDFS-8896
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Walter Su
>Assignee: Walter Su
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HDFS-8896.01.patch, screenshot_1.PNG, screenshot_2.PNG
>
>
> The anonymous {{Thread}} object created in {{ShutdownHookManager}} is a GC
> root.
> screenshot_1 shows how DN object be traced to the GC root.
> It's not a problem in production.
> It's a problem in test, especially when MiniDFSCluster starts/shutdowns many
> DNs, which could cause {{OutOfMemoryError}}.
> screenshot_2 shows many DN objects are not GCed when run the test of
> HDFS-8838.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-8968) New benchmark throughput tool for striping erasure coding
Kai Zheng created HDFS-8968: --- Summary: New benchmark throughput tool for striping erasure coding Key: HDFS-8968 URL: https://issues.apache.org/jira/browse/HDFS-8968 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Kai Zheng Assignee: Rui Li We need a new benchmark tool to measure the throughput of client writing and reading considering cases or factors: * 3-replica or striping; * write or read, stateful read or positional read; * which erasure coder; * striping cell size; * concurrent readers/writers using processes or threads. The tool should be easy to use and better to avoid unnecessary local environment impact, like local disk. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-2390) dfsadmin -setBalancerBandwidth doesnot validate -ve value
[
https://issues.apache.org/jira/browse/HDFS-2390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14715945#comment-14715945
]
Hadoop QA commented on HDFS-2390:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 17m 46s | Pre-patch trunk has 4 extant
Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 52s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 50s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 24s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 22s | The applied patch generated 1
new checkstyle issues (total was 228, now 228). |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 27s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 28s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 6s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 82m 35s | Tests failed in hadoop-hdfs. |
| | | 127m 27s | |
\\
\\
|| Reason || Tests ||
| Timed out tests | org.apache.hadoop.hdfs.server.namenode.TestINodeFile |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12752602/HDFS-2390-4.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f44b599 |
| Pre-patch Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/12148/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/12148/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12148/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/12148/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12148/console |
This message was automatically generated.
> dfsadmin -setBalancerBandwidth doesnot validate -ve value
> -
>
> Key: HDFS-2390
> URL: https://issues.apache.org/jira/browse/HDFS-2390
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer & mover
>Affects Versions: 2.7.1
>Reporter: Rajit Saha
>Assignee: Gautam Gopalakrishnan
> Attachments: HDFS-2390-1.patch, HDFS-2390-2.patch, HDFS-2390-3.patch,
> HDFS-2390-4.patch
>
>
> $ hadoop dfsadmin -setBalancerBandwidth -1
> does not throw any message that it is invalid although in DN log we are not
> getting
> "DNA_BALANCERBANDWIDTHUPDATE".
> I think it should throw some message that -ve numbers are not valid , as it
> does
> for decimal numbers or non-numbers like -
> $ hadoop dfsadmin -setBalancerBandwidth 12.34
> NumberFormatException: For input string: "12.34"
> Usage: java DFSAdmin [-setBalancerBandwidth ]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8940) Support for large-scale multi-tenant inotify service
[ https://issues.apache.org/jira/browse/HDFS-8940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14715919#comment-14715919 ] Ajith S commented on HDFS-8940: --- Hi all, Design document suggests polling from SbNN(without inotify). I think as [~cmccabe] and [~surendrasingh] mentioned "read-inotify-from-standby" also seems to be good approach, but we may also have to consider "delayed namespace information" in SbNN [~mingma] thanks for the design > Support for large-scale multi-tenant inotify service > > > Key: HDFS-8940 > URL: https://issues.apache.org/jira/browse/HDFS-8940 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ming Ma > Attachments: Large-Scale-Multi-Tenant-Inotify-Service.pdf > > > HDFS-6634 provides the core inotify functionality. We would like to extend > that to provide a large-scale service that ten of thousands of clients can > subscribe to. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8961) Investigate lock issue in o.a.h.hdfs.shortcircuit.DfsClientShmManager.EndpointShmManager
[
https://issues.apache.org/jira/browse/HDFS-8961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14715903#comment-14715903
]
Hadoop QA commented on HDFS-8961:
-
\\
\\
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 0m 0s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | release audit | 0m 15s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| | | 0m 19s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12752630/HDFS-8961.000.patch |
| Optional Tests | |
| git revision | trunk / 4cbbfa2 |
| Java | 1.7.0_55 |
| uname | Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12152/console |
This message was automatically generated.
> Investigate lock issue in
> o.a.h.hdfs.shortcircuit.DfsClientShmManager.EndpointShmManager
>
>
> Key: HDFS-8961
> URL: https://issues.apache.org/jira/browse/HDFS-8961
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Haohui Mai
>Assignee: Mingliang Liu
> Attachments: HDFS-8961.000.patch
>
>
> There are two clauses in {{hadoop-hdfs}} to filter out the findbugs warnings
> in
> {{org.apache.hadoop.hdfs.shortcircuit.DfsClientShmManager$EndpointShmManager}}:
> {code}
>
>name="org.apache.hadoop.hdfs.shortcircuit.DfsClientShmManager$EndpointShmManager"
> />
>
>
>
>
>name="org.apache.hadoop.hdfs.shortcircuit.DfsClientShmManager$EndpointShmManager"
> />
>
>
>
> {code}
> These two warnings show up in the Jenkins run as these classes are moved into
> the {{hadoop-hdfs-client}} module. We either need to fix the code or move
> these clauses to the {{hadoop-hdfs-client}} module.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8961) Investigate lock issue in o.a.h.hdfs.shortcircuit.DfsClientShmManager.EndpointShmManager
[
https://issues.apache.org/jira/browse/HDFS-8961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HDFS-8961:
Target Version/s: 2.8.0
Status: Patch Available (was: Open)
> Investigate lock issue in
> o.a.h.hdfs.shortcircuit.DfsClientShmManager.EndpointShmManager
>
>
> Key: HDFS-8961
> URL: https://issues.apache.org/jira/browse/HDFS-8961
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Haohui Mai
>Assignee: Mingliang Liu
> Attachments: HDFS-8961.000.patch
>
>
> There are two clauses in {{hadoop-hdfs}} to filter out the findbugs warnings
> in
> {{org.apache.hadoop.hdfs.shortcircuit.DfsClientShmManager$EndpointShmManager}}:
> {code}
>
>name="org.apache.hadoop.hdfs.shortcircuit.DfsClientShmManager$EndpointShmManager"
> />
>
>
>
>
>name="org.apache.hadoop.hdfs.shortcircuit.DfsClientShmManager$EndpointShmManager"
> />
>
>
>
> {code}
> These two warnings show up in the Jenkins run as these classes are moved into
> the {{hadoop-hdfs-client}} module. We either need to fix the code or move
> these clauses to the {{hadoop-hdfs-client}} module.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8900) Compact XAttrs to optimize memory footprint.
[
https://issues.apache.org/jira/browse/HDFS-8900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14715900#comment-14715900
]
Yi Liu commented on HDFS-8900:
--
Yes, Thanks Haohui.
Seems there is few issue about the Jenkins report, I can't find them in the
report of current JIRA, but we do can find them in other JIRA after it's
committed.
> Compact XAttrs to optimize memory footprint.
>
>
> Key: HDFS-8900
> URL: https://issues.apache.org/jira/browse/HDFS-8900
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Yi Liu
>Assignee: Yi Liu
> Fix For: 2.8.0
>
> Attachments: HDFS-8900.001.patch, HDFS-8900.002.patch,
> HDFS-8900.003.patch, HDFS-8900.004.patch, HDFS-8900.005.patch
>
>
> {code}
> private final ImmutableList xAttrs;
> {code}
> Currently we use above in XAttrFeature, it's not efficient from memory point
> of view, since {{ImmutableList}} and {{XAttr}} have object memory overhead,
> and each object has memory alignment.
> We can use a {{byte[]}} in XAttrFeature and do some compact in {{XAttr}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8961) Investigate lock issue in o.a.h.hdfs.shortcircuit.DfsClientShmManager.EndpointShmManager
[
https://issues.apache.org/jira/browse/HDFS-8961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HDFS-8961:
Attachment: HDFS-8961.000.patch
The v0 patch moves the findbugs filter rule from {{hadoop-hdfs}} to
{{hadoop-hdfs-client}} module.
> Investigate lock issue in
> o.a.h.hdfs.shortcircuit.DfsClientShmManager.EndpointShmManager
>
>
> Key: HDFS-8961
> URL: https://issues.apache.org/jira/browse/HDFS-8961
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Haohui Mai
>Assignee: Mingliang Liu
> Attachments: HDFS-8961.000.patch
>
>
> There are two clauses in {{hadoop-hdfs}} to filter out the findbugs warnings
> in
> {{org.apache.hadoop.hdfs.shortcircuit.DfsClientShmManager$EndpointShmManager}}:
> {code}
>
>name="org.apache.hadoop.hdfs.shortcircuit.DfsClientShmManager$EndpointShmManager"
> />
>
>
>
>
>name="org.apache.hadoop.hdfs.shortcircuit.DfsClientShmManager$EndpointShmManager"
> />
>
>
>
> {code}
> These two warnings show up in the Jenkins run as these classes are moved into
> the {{hadoop-hdfs-client}} module. We either need to fix the code or move
> these clauses to the {{hadoop-hdfs-client}} module.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8967) Create a BlockManagerLock class to represent the lock used in the BlockManager
[
https://issues.apache.org/jira/browse/HDFS-8967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haohui Mai updated HDFS-8967:
-
Status: Patch Available (was: Open)
> Create a BlockManagerLock class to represent the lock used in the BlockManager
> --
>
> Key: HDFS-8967
> URL: https://issues.apache.org/jira/browse/HDFS-8967
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Attachments: HDFS-8967.000.patch
>
>
> This jira proposes to create a {{BlockManagerLock}} class to represent the
> lock used in {{BlockManager}}.
> Currently it directly points to the {{FSNamesystem}} lock thus there are no
> functionality changes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8965) Harden edit log reading code against out of memory errors
[ https://issues.apache.org/jira/browse/HDFS-8965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-8965: --- Attachment: HDFS-8965.003.patch Fix compilation error > Harden edit log reading code against out of memory errors > - > > Key: HDFS-8965 > URL: https://issues.apache.org/jira/browse/HDFS-8965 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 2.0.0-alpha >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Attachments: HDFS-8965.001.patch, HDFS-8965.002.patch, > HDFS-8965.003.patch > > > We should harden the edit log reading code against out of memory errors. Now > that each op has a length prefix and a checksum, we can validate the checksum > before trying to load the Op data. This should avoid out of memory errors > when trying to load garbage data as Op data. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8967) Create a BlockManagerLock class to represent the lock used in the BlockManager
[
https://issues.apache.org/jira/browse/HDFS-8967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haohui Mai updated HDFS-8967:
-
Attachment: HDFS-8967.000.patch
> Create a BlockManagerLock class to represent the lock used in the BlockManager
> --
>
> Key: HDFS-8967
> URL: https://issues.apache.org/jira/browse/HDFS-8967
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Attachments: HDFS-8967.000.patch
>
>
> This jira proposes to create a {{BlockManagerLock}} class to represent the
> lock used in {{BlockManager}}.
> Currently it directly points to the {{FSNamesystem}} lock thus there are no
> functionality changes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HDFS-8963) Fix findbugs warnings introduced in HDFS-8900
[ https://issues.apache.org/jira/browse/HDFS-8963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe reassigned HDFS-8963: -- Assignee: Colin Patrick McCabe > Fix findbugs warnings introduced in HDFS-8900 > - > > Key: HDFS-8963 > URL: https://issues.apache.org/jira/browse/HDFS-8963 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Haohui Mai >Assignee: Colin Patrick McCabe > > HDFS-8900 introduced two new findbugs warnings: > https://builds.apache.org/job/PreCommit-HDFS-Build/12120/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8823) Move replication factor into individual blocks
[
https://issues.apache.org/jira/browse/HDFS-8823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haohui Mai updated HDFS-8823:
-
Issue Type: Sub-task (was: Improvement)
Parent: HDFS-8966
> Move replication factor into individual blocks
> --
>
> Key: HDFS-8823
> URL: https://issues.apache.org/jira/browse/HDFS-8823
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Fix For: 2.8.0
>
> Attachments: HDFS-8823.000.patch, HDFS-8823.001.patch,
> HDFS-8823.002.patch, HDFS-8823.003.patch, HDFS-8823.004.patch,
> HDFS-8823.005.patch, HDFS-8823.006.patch
>
>
> This jira proposes to record the replication factor in the {{BlockInfo}}
> class. The changes have two advantages:
> * Decoupling the namespace and the block management layer. It is a
> prerequisite step to move block management off the heap or to a separate
> process.
> * Increased flexibility on replicating blocks. Currently the replication
> factors of all blocks have to be the same. The replication factors of these
> blocks are equal to the highest replication factor across all snapshots. The
> changes will allow blocks in a file to have different replication factor,
> potentially saving some space.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8801) Convert BlockInfoUnderConstruction as a feature
[
https://issues.apache.org/jira/browse/HDFS-8801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haohui Mai updated HDFS-8801:
-
Issue Type: Sub-task (was: Improvement)
Parent: HDFS-8966
> Convert BlockInfoUnderConstruction as a feature
> ---
>
> Key: HDFS-8801
> URL: https://issues.apache.org/jira/browse/HDFS-8801
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Zhe Zhang
>Assignee: Jing Zhao
> Fix For: 2.8.0
>
> Attachments: HDFS-8801.000.patch
>
>
> Per discussion under HDFS-8499, with the erasure coding feature, there will
> be 4 types of {{BlockInfo}} forming a multi-inheritance:
> {{complete+contiguous}}, {{complete+striping}}, {{UC+contiguous}},
> {{UC+striped}}. We had the same challenge with {{INodeFile}} and the solution
> was building feature classes like {{FileUnderConstructionFeature}}. This JIRA
> aims to implement the same idea on {{BlockInfo}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-8967) Create a BlockManagerLock class to represent the lock used in the BlockManager
Haohui Mai created HDFS-8967:
Summary: Create a BlockManagerLock class to represent the lock
used in the BlockManager
Key: HDFS-8967
URL: https://issues.apache.org/jira/browse/HDFS-8967
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Haohui Mai
Assignee: Haohui Mai
This jira proposes to create a {{BlockManagerLock}} class to represent the lock
used in {{BlockManager}}.
Currently it directly points to the {{FSNamesystem}} lock thus there are no
functionality changes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8248) Store INodeId instead of the INodeFile object in BlockInfoContiguous
[
https://issues.apache.org/jira/browse/HDFS-8248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haohui Mai updated HDFS-8248:
-
Issue Type: Sub-task (was: Improvement)
Parent: HDFS-8966
> Store INodeId instead of the INodeFile object in BlockInfoContiguous
>
>
> Key: HDFS-8248
> URL: https://issues.apache.org/jira/browse/HDFS-8248
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Fix For: 2.8.0
>
> Attachments: HDFS-8248.000.patch, HDFS-8248.001.patch,
> HDFS-8248.002.patch, HDFS-8248.003.patch, HDFS-8248.004.patch,
> HDFS-8248.005.patch, HDFS-8248.006.patch, HDFS-8248.007.patch,
> HDFS-8248.008.patch
>
>
> Currently the namespace and the block manager are tightly coupled together.
> There are two couplings in terms of implementation:
> 1. The {{BlockInfoContiguous}} stores a reference of the {{INodeFile}} that
> owns the block, so that the block manager can look up the corresponding file
> when replicating blocks, recovering from pipeline failures, etc.
> 1. The {{INodeFile}} stores {{BlockInfoContiguous}} objects that the file
> owns.
> Decoupling the namespace and the block manager allows the BM to be separated
> out from the Java heap or even as a standalone process. This jira proposes to
> remove the first coupling by storing the id of the inode instead of the
> object reference of {{INodeFile}} in the {{BlockInfoContiguous}} class.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-8966) Separate the lock used in namespace and block management layer
Haohui Mai created HDFS-8966: Summary: Separate the lock used in namespace and block management layer Key: HDFS-8966 URL: https://issues.apache.org/jira/browse/HDFS-8966 Project: Hadoop HDFS Issue Type: Improvement Reporter: Haohui Mai Assignee: Haohui Mai Currently the namespace and the block management layer share one giant lock. One consequence that we have seen more and more often is that the namespace hangs due to excessive activities from the block management layer. For example, the NN might take a couple hundred milliseconds to handle a large block report. Because the NN holds the write lock during processing the block report, all namespace requests are paused. In production we have seen these lock contentions cause long latencies and instabilities in the cluster. This umbrella jira proposes to separate the lock used by namespace and the block management layer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8961) Investigate lock issue in o.a.h.hdfs.shortcircuit.DfsClientShmManager.EndpointShmManager
[
https://issues.apache.org/jira/browse/HDFS-8961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14715888#comment-14715888
]
Mingliang Liu commented on HDFS-8961:
-
As to the code itself, I think the two findbugs warning are false positive.
While it does not follow the JSR-166 strictly, its releasing/retaking lock
action is safe in {{DfsClientShmManager$EndpointShmManager::allocSlot()}}
method.
So I prefer the other solution for this issue. I will file a patch to move the
findbugs filter rules from {{hadoop-hdfs}} to {{hadoop-hdfs-client}} module.
> Investigate lock issue in
> o.a.h.hdfs.shortcircuit.DfsClientShmManager.EndpointShmManager
>
>
> Key: HDFS-8961
> URL: https://issues.apache.org/jira/browse/HDFS-8961
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Haohui Mai
>Assignee: Mingliang Liu
>
> There are two clauses in {{hadoop-hdfs}} to filter out the findbugs warnings
> in
> {{org.apache.hadoop.hdfs.shortcircuit.DfsClientShmManager$EndpointShmManager}}:
> {code}
>
>name="org.apache.hadoop.hdfs.shortcircuit.DfsClientShmManager$EndpointShmManager"
> />
>
>
>
>
>name="org.apache.hadoop.hdfs.shortcircuit.DfsClientShmManager$EndpointShmManager"
> />
>
>
>
> {code}
> These two warnings show up in the Jenkins run as these classes are moved into
> the {{hadoop-hdfs-client}} module. We either need to fix the code or move
> these clauses to the {{hadoop-hdfs-client}} module.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8833) Erasure coding: store EC schema and cell size in INodeFile and eliminate notion of EC zones
[
https://issues.apache.org/jira/browse/HDFS-8833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14715880#comment-14715880
]
Andrew Wang commented on HDFS-8833:
---
Looking at other distributed filesystems, many are mostly or entirely erasure
coded. QFS, Isilon OneFS, Colossus, many blobstores. I can easily see the ratio
of EC:repl being reversed, 10:1 if anything. This makes sense when you consider
data temperature; probably <10% of files are hot, so the other 90% are suitable
for EC.
I find the above usecase very compelling, which is why I've been advocating for
using the file header bits. I haven't seen much competition for the bits
either, and we can also start conservatively when using bits (only as many as
we need).
> Erasure coding: store EC schema and cell size in INodeFile and eliminate
> notion of EC zones
> ---
>
> Key: HDFS-8833
> URL: https://issues.apache.org/jira/browse/HDFS-8833
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: HDFS-7285
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Attachments: HDFS-8833-HDFS-7285-merge.00.patch,
> HDFS-8833-HDFS-7285-merge.01.patch, HDFS-8833-HDFS-7285.02.patch
>
>
> We have [discussed |
> https://issues.apache.org/jira/browse/HDFS-7285?focusedCommentId=14357754&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14357754]
> storing EC schema with files instead of EC zones and recently revisited the
> discussion under HDFS-8059.
> As a recap, the _zone_ concept has severe limitations including renaming and
> nested configuration. Those limitations are valid in encryption for security
> reasons and it doesn't make sense to carry them over in EC.
> This JIRA aims to store EC schema and cell size on {{INodeFile}} level. For
> simplicity, we should first implement it as an xattr and consider memory
> optimizations (such as moving it to file header) as a follow-on. We should
> also disable changing EC policy on a non-empty file / dir in the first phase.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8248) Store INodeId instead of the INodeFile object in BlockInfoContiguous
[
https://issues.apache.org/jira/browse/HDFS-8248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14715878#comment-14715878
]
Hudson commented on HDFS-8248:
--
FAILURE: Integrated in Hadoop-trunk-Commit #8352 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/8352/])
HDFS-8248. Store INodeId instead of the INodeFile object in
BlockInfoContiguous. Contributed by Haohui Mai. (wheat9: rev
4cbbfa2220e884e91bf18ad1cc2f3b11f895f8c9)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockCollection.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestSnapshotBlocksMap.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlocksMap.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockInfo.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestSnapshotDeletion.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCommitBlockSynchronization.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeId.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/SequentialBlockIdGenerator.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfo.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/Namesystem.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DecommissionManager.java
> Store INodeId instead of the INodeFile object in BlockInfoContiguous
>
>
> Key: HDFS-8248
> URL: https://issues.apache.org/jira/browse/HDFS-8248
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Fix For: 2.8.0
>
> Attachments: HDFS-8248.000.patch, HDFS-8248.001.patch,
> HDFS-8248.002.patch, HDFS-8248.003.patch, HDFS-8248.004.patch,
> HDFS-8248.005.patch, HDFS-8248.006.patch, HDFS-8248.007.patch,
> HDFS-8248.008.patch
>
>
> Currently the namespace and the block manager are tightly coupled together.
> There are two couplings in terms of implementation:
> 1. The {{BlockInfoContiguous}} stores a reference of the {{INodeFile}} that
> owns the block, so that the block manager can look up the corresponding file
> when replicating blocks, recovering from pipeline failures, etc.
> 1. The {{INodeFile}} stores {{BlockInfoContiguous}} objects that the file
> owns.
> Decoupling the namespace and the block manager allows the BM to be separated
> out from the Java heap or even as a standalone process. This jira proposes to
> remove the first coupling by storing the id of the inode instead of the
> object reference of {{INodeFile}} in the {{BlockInfoContiguous}} class.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8287) DFSStripedOutputStream.writeChunk should not wait for writing parity
[
https://issues.apache.org/jira/browse/HDFS-8287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14715873#comment-14715873
]
Hadoop QA commented on HDFS-8287:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 17m 53s | Findbugs (version ) appears to
be broken on HDFS-7285. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | javac | 9m 21s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 55s | There were no new javadoc
warning messages. |
| {color:red}-1{color} | release audit | 0m 15s | The applied patch generated
1 release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 37s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 32s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 35s | The patch built with
eclipse:eclipse. |
| {color:red}-1{color} | findbugs | 2m 48s | The patch appears to introduce 6
new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 55s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 142m 59s | Tests failed in hadoop-hdfs. |
| | | 189m 54s | |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs |
| Failed unit tests |
hadoop.hdfs.server.namenode.TestAddOverReplicatedStripedBlocks |
| Timed out tests | org.apache.hadoop.hdfs.TestRecoverStripedFile |
| |
org.apache.hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerWithStripedBlocks
|
| | org.apache.hadoop.hdfs.TestReadStripedFileWithMissingBlocks |
| | org.apache.hadoop.hdfs.TestBlockReaderFactory |
| | org.apache.hadoop.hdfs.TestSafeModeWithStripedFile |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12751371/HDFS-8287-HDFS-7285.05.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | HDFS-7285 / 6b6a63b |
| Release Audit |
https://builds.apache.org/job/PreCommit-HDFS-Build/12143/artifact/patchprocess/patchReleaseAuditProblems.txt
|
| Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/12143/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12143/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/12143/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12143/console |
This message was automatically generated.
> DFSStripedOutputStream.writeChunk should not wait for writing parity
> -
>
> Key: HDFS-8287
> URL: https://issues.apache.org/jira/browse/HDFS-8287
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Kai Sasaki
> Attachments: HDFS-8287-HDFS-7285.00.patch,
> HDFS-8287-HDFS-7285.01.patch, HDFS-8287-HDFS-7285.02.patch,
> HDFS-8287-HDFS-7285.03.patch, HDFS-8287-HDFS-7285.04.patch,
> HDFS-8287-HDFS-7285.05.patch
>
>
> When a stripping cell is full, writeChunk computes and generates parity
> packets. It sequentially calls waitAndQueuePacket so that user client cannot
> continue to write data until it finishes.
> We should allow user client to continue writing instead but not blocking it
> when writing parity.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8838) Erasure Coding: Tolerate datanode failures in DFSStripedOutputStream when the data length is small
[ https://issues.apache.org/jira/browse/HDFS-8838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14715868#comment-14715868 ] Tsz Wo Nicholas Sze commented on HDFS-8838: --- Thanks a lot for reviewing and committing the patch. I will continue to working on TestDFSStripedOutputStreamWithFailure 020 - 210. > Erasure Coding: Tolerate datanode failures in DFSStripedOutputStream when the > data length is small > -- > > Key: HDFS-8838 > URL: https://issues.apache.org/jira/browse/HDFS-8838 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze > Fix For: HDFS-7285 > > Attachments: HDFS-8838-HDFS-7285-000.patch, > HDFS-8838-HDFS-7285-20150809-test.patch, HDFS-8838-HDFS-7285-20150809.patch, > HDFS-8838-HDFS-7285-20150821.patch, h8838_20150729.patch, > h8838_20150731-HDFS-7285.patch, h8838_20150731.log, h8838_20150731.patch, > h8838_20150804-HDFS-7285.patch, h8838_20150809.patch > > > Currently, DFSStripedOutputStream cannot tolerate datanode failures when the > data length is small. We fix the bugs here and add more tests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8248) Store INodeId instead of the INodeFile object in BlockInfoContiguous
[
https://issues.apache.org/jira/browse/HDFS-8248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haohui Mai updated HDFS-8248:
-
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 2.8.0
Status: Resolved (was: Patch Available)
I've committed the patch to trunk and branch-2. Thanks Jing for the review.
> Store INodeId instead of the INodeFile object in BlockInfoContiguous
>
>
> Key: HDFS-8248
> URL: https://issues.apache.org/jira/browse/HDFS-8248
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Fix For: 2.8.0
>
> Attachments: HDFS-8248.000.patch, HDFS-8248.001.patch,
> HDFS-8248.002.patch, HDFS-8248.003.patch, HDFS-8248.004.patch,
> HDFS-8248.005.patch, HDFS-8248.006.patch, HDFS-8248.007.patch,
> HDFS-8248.008.patch
>
>
> Currently the namespace and the block manager are tightly coupled together.
> There are two couplings in terms of implementation:
> 1. The {{BlockInfoContiguous}} stores a reference of the {{INodeFile}} that
> owns the block, so that the block manager can look up the corresponding file
> when replicating blocks, recovering from pipeline failures, etc.
> 1. The {{INodeFile}} stores {{BlockInfoContiguous}} objects that the file
> owns.
> Decoupling the namespace and the block manager allows the BM to be separated
> out from the Java heap or even as a standalone process. This jira proposes to
> remove the first coupling by storing the id of the inode instead of the
> object reference of {{INodeFile}} in the {{BlockInfoContiguous}} class.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8248) Store INodeId instead of the INodeFile object in BlockInfoContiguous
[
https://issues.apache.org/jira/browse/HDFS-8248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14715863#comment-14715863
]
Jing Zhao commented on HDFS-8248:
-
The latest patch looks good to me. +1. The failed test passed in my local
machine.
> Store INodeId instead of the INodeFile object in BlockInfoContiguous
>
>
> Key: HDFS-8248
> URL: https://issues.apache.org/jira/browse/HDFS-8248
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Attachments: HDFS-8248.000.patch, HDFS-8248.001.patch,
> HDFS-8248.002.patch, HDFS-8248.003.patch, HDFS-8248.004.patch,
> HDFS-8248.005.patch, HDFS-8248.006.patch, HDFS-8248.007.patch,
> HDFS-8248.008.patch
>
>
> Currently the namespace and the block manager are tightly coupled together.
> There are two couplings in terms of implementation:
> 1. The {{BlockInfoContiguous}} stores a reference of the {{INodeFile}} that
> owns the block, so that the block manager can look up the corresponding file
> when replicating blocks, recovering from pipeline failures, etc.
> 1. The {{INodeFile}} stores {{BlockInfoContiguous}} objects that the file
> owns.
> Decoupling the namespace and the block manager allows the BM to be separated
> out from the Java heap or even as a standalone process. This jira proposes to
> remove the first coupling by storing the id of the inode instead of the
> object reference of {{INodeFile}} in the {{BlockInfoContiguous}} class.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8838) Erasure Coding: Tolerate datanode failures in DFSStripedOutputStream when the data length is small
[ https://issues.apache.org/jira/browse/HDFS-8838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Walter Su updated HDFS-8838: Resolution: Fixed Fix Version/s: HDFS-7285 Status: Resolved (was: Patch Available) I've committed it. Thanks [~szetszwo] for contribution. Also thanks [~jingzhao], [~zhz], [~libo-intel] for review. > Erasure Coding: Tolerate datanode failures in DFSStripedOutputStream when the > data length is small > -- > > Key: HDFS-8838 > URL: https://issues.apache.org/jira/browse/HDFS-8838 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze > Fix For: HDFS-7285 > > Attachments: HDFS-8838-HDFS-7285-000.patch, > HDFS-8838-HDFS-7285-20150809-test.patch, HDFS-8838-HDFS-7285-20150809.patch, > HDFS-8838-HDFS-7285-20150821.patch, h8838_20150729.patch, > h8838_20150731-HDFS-7285.patch, h8838_20150731.log, h8838_20150731.patch, > h8838_20150804-HDFS-7285.patch, h8838_20150809.patch > > > Currently, DFSStripedOutputStream cannot tolerate datanode failures when the > data length is small. We fix the bugs here and add more tests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8838) Erasure Coding: Tolerate datanode failures in DFSStripedOutputStream when the data length is small
[ https://issues.apache.org/jira/browse/HDFS-8838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Walter Su updated HDFS-8838: Summary: Erasure Coding: Tolerate datanode failures in DFSStripedOutputStream when the data length is small (was: Tolerate datanode failures in DFSStripedOutputStream when the data length is small) > Erasure Coding: Tolerate datanode failures in DFSStripedOutputStream when the > data length is small > -- > > Key: HDFS-8838 > URL: https://issues.apache.org/jira/browse/HDFS-8838 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze > Attachments: HDFS-8838-HDFS-7285-000.patch, > HDFS-8838-HDFS-7285-20150809-test.patch, HDFS-8838-HDFS-7285-20150809.patch, > HDFS-8838-HDFS-7285-20150821.patch, h8838_20150729.patch, > h8838_20150731-HDFS-7285.patch, h8838_20150731.log, h8838_20150731.patch, > h8838_20150804-HDFS-7285.patch, h8838_20150809.patch > > > Currently, DFSStripedOutputStream cannot tolerate datanode failures when the > data length is small. We fix the bugs here and add more tests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-3059) ssl-server.xml causes NullPointer
[
https://issues.apache.org/jira/browse/HDFS-3059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ravi Prakash updated HDFS-3059:
---
Attachment: HDFS-3059.03.patch
Here's a patch but it doesn't throw an exception. I think we should throw an
exception though :(
> ssl-server.xml causes NullPointer
> -
>
> Key: HDFS-3059
> URL: https://issues.apache.org/jira/browse/HDFS-3059
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, security
>Affects Versions: 2.7.1
> Environment: in core-site.xml:
> {code:xml}
>
> hadoop.security.authentication
> kerberos
>
>
> hadoop.security.authorization
> true
>
> {code}
> in hdfs-site.xml:
> {code:xml}
>
> dfs.https.server.keystore.resource
> /etc/hadoop/conf/ssl-server.xml
>
>
> dfs.https.enable
> true
>
>
> ...other security props
>
> {code}
>Reporter: Evert Lammerts
>Priority: Minor
> Labels: BB2015-05-TBR
> Attachments: HDFS-3059.02.patch, HDFS-3059.03.patch, HDFS-3059.patch,
> HDFS-3059.patch.2
>
>
> If ssl is enabled (dfs.https.enable) but ssl-server.xml is not available, a
> DN will crash during startup while setting up an SSL socket with a
> NullPointerException:
> {noformat}12/03/07 17:08:36 DEBUG security.Krb5AndCertsSslSocketConnector:
> useKerb = false, useCerts = true
> jetty.ssl.password : jetty.ssl.keypassword : 12/03/07 17:08:36 INFO
> mortbay.log: jetty-6.1.26.cloudera.1
> 12/03/07 17:08:36 INFO mortbay.log: Started
> selectchannelconnec...@p-worker35.alley.sara.nl:1006
> 12/03/07 17:08:36 DEBUG security.Krb5AndCertsSslSocketConnector: Creating new
> KrbServerSocket for: 0.0.0.0
> 12/03/07 17:08:36 WARN mortbay.log: java.lang.NullPointerException
> 12/03/07 17:08:36 WARN mortbay.log: failed
> Krb5AndCertsSslSocketConnector@0.0.0.0:50475: java.io.IOException:
> !JsseListener: java.lang.NullPointerException
> 12/03/07 17:08:36 WARN mortbay.log: failed Server@604788d5:
> java.io.IOException: !JsseListener: java.lang.NullPointerException
> 12/03/07 17:08:36 INFO mortbay.log: Stopped
> Krb5AndCertsSslSocketConnector@0.0.0.0:50475
> 12/03/07 17:08:36 INFO mortbay.log: Stopped
> selectchannelconnec...@p-worker35.alley.sara.nl:1006
> 12/03/07 17:08:37 INFO datanode.DataNode: Waiting for threadgroup to exit,
> active threads is 0{noformat}
> The same happens if I set an absolute path to an existing
> dfs.https.server.keystore.resource - in this case the file cannot be found
> but not even a WARN is given.
> Since in dfs.https.server.keystore.resource we know we need to have 4
> properties specified (ssl.server.truststore.location,
> ssl.server.keystore.location, ssl.server.keystore.password, and
> ssl.server.keystore.keypassword) we should check if they are set and throw an
> IOException if they are not.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8909) Erasure coding: update BlockInfoContiguousUC and BlockInfoStripedUC to use BlockUnderConstructionFeature
[
https://issues.apache.org/jira/browse/HDFS-8909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jing Zhao updated HDFS-8909:
Attachment: HDFS-8909-HDFS-7285.004.patch
Thanks for the analysis of the test failure, Walter! Update the patch based on
your analysis to fix TestFileAppend4. The change is
{code}
-LocatedBlock lBlk = BlockManager.newLocatedBlock(fsn.getExtendedBlock(blk),
-blk, locs, offset);
+LocatedBlock lBlk = BlockManager.newLocatedBlock(
+fsn.getExtendedBlock(new Block(blk)), blk, locs, offset);
{code}
> Erasure coding: update BlockInfoContiguousUC and BlockInfoStripedUC to use
> BlockUnderConstructionFeature
>
>
> Key: HDFS-8909
> URL: https://issues.apache.org/jira/browse/HDFS-8909
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: HDFS-7285
>Reporter: Zhe Zhang
>Assignee: Jing Zhao
> Attachments: HDFS-8909-HDFS-7285.003.patch,
> HDFS-8909-HDFS-7285.004.patch, HDFS-8909.000.patch, HDFS-8909.001.patch,
> HDFS-8909.002.patch, HDFS-8909.003.patch
>
>
> HDFS-8801 converts {{BlockInfoUC}} as a feature. We should consolidate
> {{BlockInfoContiguousUC}} and {{BlockInfoStripedUC}} logics to use this
> feature.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8896) DataNode object isn't GCed when shutdown, because it has GC root in ShutdownHookManager
[
https://issues.apache.org/jira/browse/HDFS-8896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14715848#comment-14715848
]
Hudson commented on HDFS-8896:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk #2256 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2256/])
HDFS-8896. DataNode object isn't GCed when shutdown, because it has GC root in
ShutdownHookManager. Contributed by Walter Su. (jing9: rev
f44b599003bb79b1ec9b92e7546546523ec01676)
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java
*
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ShutdownHookManager.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
> DataNode object isn't GCed when shutdown, because it has GC root in
> ShutdownHookManager
> ---
>
> Key: HDFS-8896
> URL: https://issues.apache.org/jira/browse/HDFS-8896
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Walter Su
>Assignee: Walter Su
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HDFS-8896.01.patch, screenshot_1.PNG, screenshot_2.PNG
>
>
> The anonymous {{Thread}} object created in {{ShutdownHookManager}} is a GC
> root.
> screenshot_1 shows how DN object be traced to the GC root.
> It's not a problem in production.
> It's a problem in test, especially when MiniDFSCluster starts/shutdowns many
> DNs, which could cause {{OutOfMemoryError}}.
> screenshot_2 shows many DN objects are not GCed when run the test of
> HDFS-8838.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8838) Tolerate datanode failures in DFSStripedOutputStream when the data length is small
[ https://issues.apache.org/jira/browse/HDFS-8838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14715843#comment-14715843 ] Walter Su commented on HDFS-8838: - Thanks [~jingzhao]. I'll commit this shortly. > Tolerate datanode failures in DFSStripedOutputStream when the data length is > small > -- > > Key: HDFS-8838 > URL: https://issues.apache.org/jira/browse/HDFS-8838 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze > Attachments: HDFS-8838-HDFS-7285-000.patch, > HDFS-8838-HDFS-7285-20150809-test.patch, HDFS-8838-HDFS-7285-20150809.patch, > HDFS-8838-HDFS-7285-20150821.patch, h8838_20150729.patch, > h8838_20150731-HDFS-7285.patch, h8838_20150731.log, h8838_20150731.patch, > h8838_20150804-HDFS-7285.patch, h8838_20150809.patch > > > Currently, DFSStripedOutputStream cannot tolerate datanode failures when the > data length is small. We fix the bugs here and add more tests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8951) Move the shortcircuit package to hdfs-client
[
https://issues.apache.org/jira/browse/HDFS-8951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14715835#comment-14715835
]
Hudson commented on HDFS-8951:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk #2236 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/2236/])
HDFS-8951. Move the shortcircuit package to hdfs-client. Contributed by
Mingliang Liu. (wheat9: rev c992bcf9c136d3df686655a80e636bb7bb0664da)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/shortcircuit/DomainSocketFactory.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/shortcircuit/ShortCircuitCache.java
*
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/shortcircuit/DomainSocketFactory.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/shortcircuit/ClientMmap.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderFactory.java
*
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/shortcircuit/ShortCircuitReplicaInfo.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/RemoteBlockReader2.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/RemoteBlockReader.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/shortcircuit/ShortCircuitReplicaInfo.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
*
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/shortcircuit/ShortCircuitCache.java
*
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/util/IOUtilsClient.java
*
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/shortcircuit/ClientMmap.java
*
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSUtilClient.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/RamDiskAsyncLazyPersistService.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockMetadataHeader.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/shortcircuit/ShortCircuitReplica.java
*
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/shortcircuit/ShortCircuitReplica.java
*
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockMetadataHeader.java
> Move the shortcircuit package to hdfs-client
>
>
> Key: HDFS-8951
> URL: https://issues.apache.org/jira/browse/HDFS-8951
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: build
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Fix For: 2.8.0
>
> Attachments: HDFS-8951.000.patch, HDFS-8951.001.patch,
> HDFS-8951.002.patch
>
>
> This jira tracks the effort of moving the {{shortcircuit}} package into the
> hdfs-client module.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-3059) ssl-server.xml causes NullPointer
[
https://issues.apache.org/jira/browse/HDFS-3059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ravi Prakash updated HDFS-3059:
---
Affects Version/s: (was: 1.0.0)
(was: 0.20.205.0)
2.7.1
Target Version/s: 3.0.0 (was: 0.20.205.0)
Status: Open (was: Patch Available)
Patch doesn't apply on trunk anymore :(
> ssl-server.xml causes NullPointer
> -
>
> Key: HDFS-3059
> URL: https://issues.apache.org/jira/browse/HDFS-3059
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, security
>Affects Versions: 2.7.1
> Environment: in core-site.xml:
> {code:xml}
>
> hadoop.security.authentication
> kerberos
>
>
> hadoop.security.authorization
> true
>
> {code}
> in hdfs-site.xml:
> {code:xml}
>
> dfs.https.server.keystore.resource
> /etc/hadoop/conf/ssl-server.xml
>
>
> dfs.https.enable
> true
>
>
> ...other security props
>
> {code}
>Reporter: Evert Lammerts
>Priority: Minor
> Labels: BB2015-05-TBR
> Attachments: HDFS-3059.02.patch, HDFS-3059.patch, HDFS-3059.patch.2
>
>
> If ssl is enabled (dfs.https.enable) but ssl-server.xml is not available, a
> DN will crash during startup while setting up an SSL socket with a
> NullPointerException:
> {noformat}12/03/07 17:08:36 DEBUG security.Krb5AndCertsSslSocketConnector:
> useKerb = false, useCerts = true
> jetty.ssl.password : jetty.ssl.keypassword : 12/03/07 17:08:36 INFO
> mortbay.log: jetty-6.1.26.cloudera.1
> 12/03/07 17:08:36 INFO mortbay.log: Started
> selectchannelconnec...@p-worker35.alley.sara.nl:1006
> 12/03/07 17:08:36 DEBUG security.Krb5AndCertsSslSocketConnector: Creating new
> KrbServerSocket for: 0.0.0.0
> 12/03/07 17:08:36 WARN mortbay.log: java.lang.NullPointerException
> 12/03/07 17:08:36 WARN mortbay.log: failed
> Krb5AndCertsSslSocketConnector@0.0.0.0:50475: java.io.IOException:
> !JsseListener: java.lang.NullPointerException
> 12/03/07 17:08:36 WARN mortbay.log: failed Server@604788d5:
> java.io.IOException: !JsseListener: java.lang.NullPointerException
> 12/03/07 17:08:36 INFO mortbay.log: Stopped
> Krb5AndCertsSslSocketConnector@0.0.0.0:50475
> 12/03/07 17:08:36 INFO mortbay.log: Stopped
> selectchannelconnec...@p-worker35.alley.sara.nl:1006
> 12/03/07 17:08:37 INFO datanode.DataNode: Waiting for threadgroup to exit,
> active threads is 0{noformat}
> The same happens if I set an absolute path to an existing
> dfs.https.server.keystore.resource - in this case the file cannot be found
> but not even a WARN is given.
> Since in dfs.https.server.keystore.resource we know we need to have 4
> properties specified (ssl.server.truststore.location,
> ssl.server.keystore.location, ssl.server.keystore.password, and
> ssl.server.keystore.keypassword) we should check if they are set and throw an
> IOException if they are not.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8925) Move BlockReader to hdfs-client
[
https://issues.apache.org/jira/browse/HDFS-8925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14715825#comment-14715825
]
Hadoop QA commented on HDFS-8925:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 20m 45s | Pre-patch trunk has 6 extant
Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 5 new or modified test files. |
| {color:red}-1{color} | javac | 7m 51s | The applied patch generated 16
additional warning messages. |
| {color:green}+1{color} | javadoc | 10m 5s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 25s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 2m 30s | The applied patch generated
101 new checkstyle issues (total was 90, now 191). |
| {color:green}+1{color} | whitespace | 0m 3s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 37s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 4m 35s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 13s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 78m 42s | Tests failed in hadoop-hdfs. |
| {color:green}+1{color} | hdfs tests | 0m 30s | Tests passed in
hadoop-hdfs-client. |
| | | 130m 55s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests |
hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForContentSummary |
| | hadoop.hdfs.TestSafeMode |
| | hadoop.hdfs.TestHFlush |
| | hadoop.hdfs.TestModTime |
| | hadoop.hdfs.server.blockmanagement.TestBlockManager |
| | hadoop.hdfs.server.datanode.TestReadOnlySharedStorage |
| | hadoop.hdfs.server.datanode.TestIncrementalBlockReports |
| | hadoop.hdfs.TestReservedRawPaths |
| | hadoop.hdfs.web.TestHttpsFileSystem |
| | hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes |
| | hadoop.hdfs.TestRemoteBlockReader2 |
| | hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation |
| | hadoop.hdfs.TestReplication |
| | hadoop.hdfs.TestBlocksScheduledCounter |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.TestDFSInotifyEventInputStream |
| | hadoop.hdfs.TestAbandonBlock |
| | hadoop.hdfs.TestSetTimes |
| | hadoop.hdfs.TestDFSFinalize |
| | hadoop.hdfs.web.TestFSMainOperationsWebHdfs |
| | hadoop.hdfs.server.datanode.TestBlockRecovery |
| | hadoop.hdfs.server.blockmanagement.TestNodeCount |
| | hadoop.hdfs.TestAppendDifferentChecksum |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestDatanodeRestart |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistFiles |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration |
| | hadoop.hdfs.TestHDFSFileSystemContract |
| | hadoop.hdfs.tools.TestDFSZKFailoverController |
| | hadoop.hdfs.server.datanode.TestNNHandlesCombinedBlockReport |
| | hadoop.hdfs.TestDFSShell |
| | hadoop.hdfs.TestMissingBlocksAlert |
| | hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes |
| | hadoop.hdfs.TestFileCreationClient |
| | hadoop.hdfs.TestClientReportBadBlock |
| | hadoop.hdfs.web.TestWebHDFS |
| | hadoop.hdfs.TestSmallBlock |
| | hadoop.hdfs.TestHdfsAdmin |
| | hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork |
| | hadoop.hdfs.TestReadWhileWriting |
| | hadoop.hdfs.web.TestWebHdfsTokens |
| | hadoop.hdfs.server.datanode.TestTransferRbw |
| | hadoop.hdfs.TestFileCreation |
| | hadoop.hdfs.TestRead |
| | hadoop.hdfs.tools.TestDebugAdmin |
| | hadoop.hdfs.crypto.TestHdfsCryptoStreams |
| | hadoop.hdfs.server.datanode.TestBlockHasMultipleReplicasOnSameDN |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestInterDatanodeProtocol |
| | hadoop.hdfs.qjournal.TestSecureNNWithQJM |
| | hadoop.hdfs.TestClientProtocolForPipelineRecovery |
| | hadoop.hdfs.server.datanode.TestDataNodeMetrics |
| | hadoop.hdfs.TestWriteConfigurationToDFS |
| | hadoop.hdfs.TestParallelShortCircuitReadNoChecksum |
| | hadoop.hdfs.server.mover.TestStorageMover |
| | hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForAcl |
| | hadoop.hdfs.protocolPB.TestPBHelper |
| | hadoop.hdfs.TestRenameWhileOpen |
| | hadoop.hdfs.TestDFSClientRetries |
| | hadoop.hdfs.TestSnapshotCommands |
| | hadoop.hdfs.tools.TestDFSAdminWithHA |
| | hadoop.hdfs.server.blockmanagement.TestReplicationPolicyConsiderLoad |
| | hadoop.hdfs.TestEncryptionZonesWithKMS |
| | hadoop.hdfs.TestFileAppend |
| | hado
[jira] [Commented] (HDFS-8951) Move the shortcircuit package to hdfs-client
[
https://issues.apache.org/jira/browse/HDFS-8951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14715823#comment-14715823
]
Hudson commented on HDFS-8951:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #298 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/298/])
HDFS-8951. Move the shortcircuit package to hdfs-client. Contributed by
Mingliang Liu. (wheat9: rev c992bcf9c136d3df686655a80e636bb7bb0664da)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderFactory.java
*
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSUtilClient.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/RemoteBlockReader2.java
*
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/shortcircuit/ShortCircuitReplicaInfo.java
*
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/shortcircuit/ShortCircuitReplica.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
*
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockMetadataHeader.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/shortcircuit/ClientMmap.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/RemoteBlockReader.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/shortcircuit/DomainSocketFactory.java
*
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/shortcircuit/ShortCircuitCache.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/shortcircuit/ShortCircuitReplicaInfo.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockMetadataHeader.java
*
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/shortcircuit/ClientMmap.java
*
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/shortcircuit/DomainSocketFactory.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/shortcircuit/ShortCircuitCache.java
*
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/util/IOUtilsClient.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/shortcircuit/ShortCircuitReplica.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/RamDiskAsyncLazyPersistService.java
> Move the shortcircuit package to hdfs-client
>
>
> Key: HDFS-8951
> URL: https://issues.apache.org/jira/browse/HDFS-8951
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: build
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Fix For: 2.8.0
>
> Attachments: HDFS-8951.000.patch, HDFS-8951.001.patch,
> HDFS-8951.002.patch
>
>
> This jira tracks the effort of moving the {{shortcircuit}} package into the
> hdfs-client module.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8896) DataNode object isn't GCed when shutdown, because it has GC root in ShutdownHookManager
[
https://issues.apache.org/jira/browse/HDFS-8896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14715814#comment-14715814
]
Hudson commented on HDFS-8896:
--
FAILURE: Integrated in Hadoop-Yarn-trunk #1040 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/1040/])
HDFS-8896. DataNode object isn't GCed when shutdown, because it has GC root in
ShutdownHookManager. Contributed by Walter Su. (jing9: rev
f44b599003bb79b1ec9b92e7546546523ec01676)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java
*
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ShutdownHookManager.java
> DataNode object isn't GCed when shutdown, because it has GC root in
> ShutdownHookManager
> ---
>
> Key: HDFS-8896
> URL: https://issues.apache.org/jira/browse/HDFS-8896
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Walter Su
>Assignee: Walter Su
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HDFS-8896.01.patch, screenshot_1.PNG, screenshot_2.PNG
>
>
> The anonymous {{Thread}} object created in {{ShutdownHookManager}} is a GC
> root.
> screenshot_1 shows how DN object be traced to the GC root.
> It's not a problem in production.
> It's a problem in test, especially when MiniDFSCluster starts/shutdowns many
> DNs, which could cause {{OutOfMemoryError}}.
> screenshot_2 shows many DN objects are not GCed when run the test of
> HDFS-8838.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8248) Store INodeId instead of the INodeFile object in BlockInfoContiguous
[
https://issues.apache.org/jira/browse/HDFS-8248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14715802#comment-14715802
]
Hadoop QA commented on HDFS-8248:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 17m 53s | Pre-patch trunk has 4 extant
Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 7 new or modified test files. |
| {color:green}+1{color} | javac | 7m 57s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 59s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 22s | The applied patch generated 3
new checkstyle issues (total was 512, now 510). |
| {color:green}+1{color} | whitespace | 0m 3s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 32s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 35s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 35s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 15s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 187m 16s | Tests failed in hadoop-hdfs. |
| | | 232m 54s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.hdfs.TestRollingUpgrade |
| | hadoop.hdfs.server.namenode.TestFileTruncate |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12752556/HDFS-8248.008.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / a4d9acc |
| Pre-patch Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/12142/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/12142/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12142/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/12142/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12142/console |
This message was automatically generated.
> Store INodeId instead of the INodeFile object in BlockInfoContiguous
>
>
> Key: HDFS-8248
> URL: https://issues.apache.org/jira/browse/HDFS-8248
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Attachments: HDFS-8248.000.patch, HDFS-8248.001.patch,
> HDFS-8248.002.patch, HDFS-8248.003.patch, HDFS-8248.004.patch,
> HDFS-8248.005.patch, HDFS-8248.006.patch, HDFS-8248.007.patch,
> HDFS-8248.008.patch
>
>
> Currently the namespace and the block manager are tightly coupled together.
> There are two couplings in terms of implementation:
> 1. The {{BlockInfoContiguous}} stores a reference of the {{INodeFile}} that
> owns the block, so that the block manager can look up the corresponding file
> when replicating blocks, recovering from pipeline failures, etc.
> 1. The {{INodeFile}} stores {{BlockInfoContiguous}} objects that the file
> owns.
> Decoupling the namespace and the block manager allows the BM to be separated
> out from the Java heap or even as a standalone process. This jira proposes to
> remove the first coupling by storing the id of the inode instead of the
> object reference of {{INodeFile}} in the {{BlockInfoContiguous}} class.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8965) Harden edit log reading code against out of memory errors
[
https://issues.apache.org/jira/browse/HDFS-8965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14715796#comment-14715796
]
Hadoop QA commented on HDFS-8965:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 17m 42s | Pre-patch trunk has 4 extant
Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:red}-1{color} | javac | 2m 13s | The patch appears to cause the
build to fail. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12752600/HDFS-8965.002.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f44b599 |
| Pre-patch Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/12147/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
|
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12147/console |
This message was automatically generated.
> Harden edit log reading code against out of memory errors
> -
>
> Key: HDFS-8965
> URL: https://issues.apache.org/jira/browse/HDFS-8965
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 2.0.0-alpha
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Attachments: HDFS-8965.001.patch, HDFS-8965.002.patch
>
>
> We should harden the edit log reading code against out of memory errors. Now
> that each op has a length prefix and a checksum, we can validate the checksum
> before trying to load the Op data. This should avoid out of memory errors
> when trying to load garbage data as Op data.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8951) Move the shortcircuit package to hdfs-client
[
https://issues.apache.org/jira/browse/HDFS-8951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14715793#comment-14715793
]
Hudson commented on HDFS-8951:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #306 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/306/])
HDFS-8951. Move the shortcircuit package to hdfs-client. Contributed by
Mingliang Liu. (wheat9: rev c992bcf9c136d3df686655a80e636bb7bb0664da)
*
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/shortcircuit/ShortCircuitCache.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/RemoteBlockReader.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/RemoteBlockReader2.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderFactory.java
*
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSUtilClient.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockMetadataHeader.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/shortcircuit/ShortCircuitCache.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/shortcircuit/ShortCircuitReplicaInfo.java
*
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockMetadataHeader.java
*
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/util/IOUtilsClient.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/shortcircuit/DomainSocketFactory.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/shortcircuit/ClientMmap.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/shortcircuit/ShortCircuitReplica.java
*
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/shortcircuit/ShortCircuitReplica.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
*
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/shortcircuit/ClientMmap.java
*
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/shortcircuit/ShortCircuitReplicaInfo.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/RamDiskAsyncLazyPersistService.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/shortcircuit/DomainSocketFactory.java
> Move the shortcircuit package to hdfs-client
>
>
> Key: HDFS-8951
> URL: https://issues.apache.org/jira/browse/HDFS-8951
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: build
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Fix For: 2.8.0
>
> Attachments: HDFS-8951.000.patch, HDFS-8951.001.patch,
> HDFS-8951.002.patch
>
>
> This jira tracks the effort of moving the {{shortcircuit}} package into the
> hdfs-client module.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8951) Move the shortcircuit package to hdfs-client
[
https://issues.apache.org/jira/browse/HDFS-8951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14715784#comment-14715784
]
Hudson commented on HDFS-8951:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk #2255 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2255/])
HDFS-8951. Move the shortcircuit package to hdfs-client. Contributed by
Mingliang Liu. (wheat9: rev c992bcf9c136d3df686655a80e636bb7bb0664da)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
*
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/shortcircuit/ShortCircuitReplica.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
*
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/shortcircuit/ShortCircuitCache.java
*
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/shortcircuit/DomainSocketFactory.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/shortcircuit/ClientMmap.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/shortcircuit/ShortCircuitCache.java
*
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/util/IOUtilsClient.java
*
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockMetadataHeader.java
*
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/shortcircuit/ShortCircuitReplicaInfo.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderFactory.java
*
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/shortcircuit/ClientMmap.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/RemoteBlockReader.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/RemoteBlockReader2.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/RamDiskAsyncLazyPersistService.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockMetadataHeader.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/shortcircuit/DomainSocketFactory.java
*
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSUtilClient.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/shortcircuit/ShortCircuitReplicaInfo.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/shortcircuit/ShortCircuitReplica.java
> Move the shortcircuit package to hdfs-client
>
>
> Key: HDFS-8951
> URL: https://issues.apache.org/jira/browse/HDFS-8951
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: build
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Fix For: 2.8.0
>
> Attachments: HDFS-8951.000.patch, HDFS-8951.001.patch,
> HDFS-8951.002.patch
>
>
> This jira tracks the effort of moving the {{shortcircuit}} package into the
> hdfs-client module.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8896) DataNode object isn't GCed when shutdown, because it has GC root in ShutdownHookManager
[
https://issues.apache.org/jira/browse/HDFS-8896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14715781#comment-14715781
]
Hudson commented on HDFS-8896:
--
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #312 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/312/])
HDFS-8896. DataNode object isn't GCed when shutdown, because it has GC root in
ShutdownHookManager. Contributed by Walter Su. (jing9: rev
f44b599003bb79b1ec9b92e7546546523ec01676)
*
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ShutdownHookManager.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
> DataNode object isn't GCed when shutdown, because it has GC root in
> ShutdownHookManager
> ---
>
> Key: HDFS-8896
> URL: https://issues.apache.org/jira/browse/HDFS-8896
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Walter Su
>Assignee: Walter Su
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HDFS-8896.01.patch, screenshot_1.PNG, screenshot_2.PNG
>
>
> The anonymous {{Thread}} object created in {{ShutdownHookManager}} is a GC
> root.
> screenshot_1 shows how DN object be traced to the GC root.
> It's not a problem in production.
> It's a problem in test, especially when MiniDFSCluster starts/shutdowns many
> DNs, which could cause {{OutOfMemoryError}}.
> screenshot_2 shows many DN objects are not GCed when run the test of
> HDFS-8838.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8965) Harden edit log reading code against out of memory errors
[
https://issues.apache.org/jira/browse/HDFS-8965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14715770#comment-14715770
]
Hadoop QA commented on HDFS-8965:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 16m 8s | Findbugs (version ) appears to
be broken on trunk. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:red}-1{color} | javac | 2m 12s | The patch appears to cause the
build to fail. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12752600/HDFS-8965.002.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f44b599 |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12145/console |
This message was automatically generated.
> Harden edit log reading code against out of memory errors
> -
>
> Key: HDFS-8965
> URL: https://issues.apache.org/jira/browse/HDFS-8965
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 2.0.0-alpha
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Attachments: HDFS-8965.001.patch, HDFS-8965.002.patch
>
>
> We should harden the edit log reading code against out of memory errors. Now
> that each op has a length prefix and a checksum, we can validate the checksum
> before trying to load the Op data. This should avoid out of memory errors
> when trying to load garbage data as Op data.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8838) Tolerate datanode failures in DFSStripedOutputStream when the data length is small
[ https://issues.apache.org/jira/browse/HDFS-8838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14715767#comment-14715767 ] Jing Zhao commented on HDFS-8838: - Hi [~walter.k.su], I've committed HDFS-8896 in trunk/branch-2. So please see how you want to commit this patch to the 7285 feature branch. > Tolerate datanode failures in DFSStripedOutputStream when the data length is > small > -- > > Key: HDFS-8838 > URL: https://issues.apache.org/jira/browse/HDFS-8838 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze > Attachments: HDFS-8838-HDFS-7285-000.patch, > HDFS-8838-HDFS-7285-20150809-test.patch, HDFS-8838-HDFS-7285-20150809.patch, > HDFS-8838-HDFS-7285-20150821.patch, h8838_20150729.patch, > h8838_20150731-HDFS-7285.patch, h8838_20150731.log, h8838_20150731.patch, > h8838_20150804-HDFS-7285.patch, h8838_20150809.patch > > > Currently, DFSStripedOutputStream cannot tolerate datanode failures when the > data length is small. We fix the bugs here and add more tests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8896) DataNode object isn't GCed when shutdown, because it has GC root in ShutdownHookManager
[
https://issues.apache.org/jira/browse/HDFS-8896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jing Zhao updated HDFS-8896:
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 2.8.0
Status: Resolved (was: Patch Available)
The patch looks good to me. +1. I've committed this to trunk and branch-2.
Thanks Walter!
> DataNode object isn't GCed when shutdown, because it has GC root in
> ShutdownHookManager
> ---
>
> Key: HDFS-8896
> URL: https://issues.apache.org/jira/browse/HDFS-8896
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Walter Su
>Assignee: Walter Su
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HDFS-8896.01.patch, screenshot_1.PNG, screenshot_2.PNG
>
>
> The anonymous {{Thread}} object created in {{ShutdownHookManager}} is a GC
> root.
> screenshot_1 shows how DN object be traced to the GC root.
> It's not a problem in production.
> It's a problem in test, especially when MiniDFSCluster starts/shutdowns many
> DNs, which could cause {{OutOfMemoryError}}.
> screenshot_2 shows many DN objects are not GCed when run the test of
> HDFS-8838.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8896) DataNode object isn't GCed when shutdown, because it has GC root in ShutdownHookManager
[
https://issues.apache.org/jira/browse/HDFS-8896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14715757#comment-14715757
]
Hudson commented on HDFS-8896:
--
FAILURE: Integrated in Hadoop-trunk-Commit #8351 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/8351/])
HDFS-8896. DataNode object isn't GCed when shutdown, because it has GC root in
ShutdownHookManager. Contributed by Walter Su. (jing9: rev
f44b599003bb79b1ec9b92e7546546523ec01676)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ShutdownHookManager.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java
> DataNode object isn't GCed when shutdown, because it has GC root in
> ShutdownHookManager
> ---
>
> Key: HDFS-8896
> URL: https://issues.apache.org/jira/browse/HDFS-8896
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Walter Su
>Assignee: Walter Su
>Priority: Minor
> Attachments: HDFS-8896.01.patch, screenshot_1.PNG, screenshot_2.PNG
>
>
> The anonymous {{Thread}} object created in {{ShutdownHookManager}} is a GC
> root.
> screenshot_1 shows how DN object be traced to the GC root.
> It's not a problem in production.
> It's a problem in test, especially when MiniDFSCluster starts/shutdowns many
> DNs, which could cause {{OutOfMemoryError}}.
> screenshot_2 shows many DN objects are not GCed when run the test of
> HDFS-8838.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-2390) dfsadmin -setBalancerBandwidth doesnot validate -ve value
[ https://issues.apache.org/jira/browse/HDFS-2390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gautam Gopalakrishnan updated HDFS-2390: Status: Patch Available (was: Open) > dfsadmin -setBalancerBandwidth doesnot validate -ve value > - > > Key: HDFS-2390 > URL: https://issues.apache.org/jira/browse/HDFS-2390 > Project: Hadoop HDFS > Issue Type: Bug > Components: balancer & mover >Affects Versions: 2.7.1 >Reporter: Rajit Saha >Assignee: Gautam Gopalakrishnan > Attachments: HDFS-2390-1.patch, HDFS-2390-2.patch, HDFS-2390-3.patch, > HDFS-2390-4.patch > > > $ hadoop dfsadmin -setBalancerBandwidth -1 > does not throw any message that it is invalid although in DN log we are not > getting > "DNA_BALANCERBANDWIDTHUPDATE". > I think it should throw some message that -ve numbers are not valid , as it > does > for decimal numbers or non-numbers like - > $ hadoop dfsadmin -setBalancerBandwidth 12.34 > NumberFormatException: For input string: "12.34" > Usage: java DFSAdmin [-setBalancerBandwidth ] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-2390) dfsadmin -setBalancerBandwidth doesnot validate -ve value
[ https://issues.apache.org/jira/browse/HDFS-2390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gautam Gopalakrishnan updated HDFS-2390: Status: Open (was: Patch Available) > dfsadmin -setBalancerBandwidth doesnot validate -ve value > - > > Key: HDFS-2390 > URL: https://issues.apache.org/jira/browse/HDFS-2390 > Project: Hadoop HDFS > Issue Type: Bug > Components: balancer & mover >Affects Versions: 2.7.1 >Reporter: Rajit Saha >Assignee: Gautam Gopalakrishnan > Attachments: HDFS-2390-1.patch, HDFS-2390-2.patch, HDFS-2390-3.patch, > HDFS-2390-4.patch > > > $ hadoop dfsadmin -setBalancerBandwidth -1 > does not throw any message that it is invalid although in DN log we are not > getting > "DNA_BALANCERBANDWIDTHUPDATE". > I think it should throw some message that -ve numbers are not valid , as it > does > for decimal numbers or non-numbers like - > $ hadoop dfsadmin -setBalancerBandwidth 12.34 > NumberFormatException: For input string: "12.34" > Usage: java DFSAdmin [-setBalancerBandwidth ] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-2390) dfsadmin -setBalancerBandwidth doesnot validate -ve value
[ https://issues.apache.org/jira/browse/HDFS-2390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gautam Gopalakrishnan updated HDFS-2390: Attachment: HDFS-2390-4.patch Fixed the message as per the review. > dfsadmin -setBalancerBandwidth doesnot validate -ve value > - > > Key: HDFS-2390 > URL: https://issues.apache.org/jira/browse/HDFS-2390 > Project: Hadoop HDFS > Issue Type: Bug > Components: balancer & mover >Affects Versions: 2.7.1 >Reporter: Rajit Saha >Assignee: Gautam Gopalakrishnan > Attachments: HDFS-2390-1.patch, HDFS-2390-2.patch, HDFS-2390-3.patch, > HDFS-2390-4.patch > > > $ hadoop dfsadmin -setBalancerBandwidth -1 > does not throw any message that it is invalid although in DN log we are not > getting > "DNA_BALANCERBANDWIDTHUPDATE". > I think it should throw some message that -ve numbers are not valid , as it > does > for decimal numbers or non-numbers like - > $ hadoop dfsadmin -setBalancerBandwidth 12.34 > NumberFormatException: For input string: "12.34" > Usage: java DFSAdmin [-setBalancerBandwidth ] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8951) Move the shortcircuit package to hdfs-client
[
https://issues.apache.org/jira/browse/HDFS-8951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14715737#comment-14715737
]
Hudson commented on HDFS-8951:
--
FAILURE: Integrated in Hadoop-Yarn-trunk #1039 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/1039/])
HDFS-8951. Move the shortcircuit package to hdfs-client. Contributed by
Mingliang Liu. (wheat9: rev c992bcf9c136d3df686655a80e636bb7bb0664da)
*
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/util/IOUtilsClient.java
*
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSUtilClient.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/RemoteBlockReader2.java
*
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockMetadataHeader.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/RemoteBlockReader.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/shortcircuit/ShortCircuitReplica.java
*
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/shortcircuit/ShortCircuitReplicaInfo.java
*
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/shortcircuit/ClientMmap.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/shortcircuit/DomainSocketFactory.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/shortcircuit/ShortCircuitReplicaInfo.java
*
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/shortcircuit/ShortCircuitReplica.java
*
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/shortcircuit/ShortCircuitCache.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockMetadataHeader.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/RamDiskAsyncLazyPersistService.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/shortcircuit/ClientMmap.java
*
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/shortcircuit/DomainSocketFactory.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderFactory.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/shortcircuit/ShortCircuitCache.java
> Move the shortcircuit package to hdfs-client
>
>
> Key: HDFS-8951
> URL: https://issues.apache.org/jira/browse/HDFS-8951
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: build
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Fix For: 2.8.0
>
> Attachments: HDFS-8951.000.patch, HDFS-8951.001.patch,
> HDFS-8951.002.patch
>
>
> This jira tracks the effort of moving the {{shortcircuit}} package into the
> hdfs-client module.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8920) Erasure Coding: when recovering lost blocks, logs can be too verbose and hurt performance
[
https://issues.apache.org/jira/browse/HDFS-8920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14715736#comment-14715736
]
Zhe Zhang commented on HDFS-8920:
-
Thanks Rui for the good thought.
bq. only print the warning when we try to get a striped block from a dead DN
for 1st time
This looks like an improved version of my suggestion above. I think we should
use this method.
> Erasure Coding: when recovering lost blocks, logs can be too verbose and hurt
> performance
> -
>
> Key: HDFS-8920
> URL: https://issues.apache.org/jira/browse/HDFS-8920
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Rui Li
>Assignee: Rui Li
>
> When we test reading data with datanodes killed,
> {{DFSInputStream::getBestNodeDNAddrPair}} becomes a hot spot method and
> effectively blocks the client JVM. This log seems too verbose:
> {code}
> if (chosenNode == null) {
> DFSClient.LOG.warn("No live nodes contain block " + block.getBlock() +
> " after checking nodes = " + Arrays.toString(nodes) +
> ", ignoredNodes = " + ignoredNodes);
> return null;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8829) DataNode sets SO_RCVBUF explicitly is disabling tcp auto-tuning
[
https://issues.apache.org/jira/browse/HDFS-8829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14715734#comment-14715734
]
Colin Patrick McCabe commented on HDFS-8829:
I understand that the {{DEFAULT_DATA_SOCKET_SIZE}} constant is currently used
for both receive buffer size and send buffer size. However, I do not think
that we should add a configuration key that acts the same way. We should have
the ability to configure receive buffer size and send buffer size separately.
> DataNode sets SO_RCVBUF explicitly is disabling tcp auto-tuning
> ---
>
> Key: HDFS-8829
> URL: https://issues.apache.org/jira/browse/HDFS-8829
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 2.3.0, 2.6.0
>Reporter: He Tianyi
>Assignee: He Tianyi
> Attachments: HDFS-8829.0001.patch, HDFS-8829.0002.patch
>
>
> {code:java}
> private void initDataXceiver(Configuration conf) throws IOException {
> // find free port or use privileged port provided
> TcpPeerServer tcpPeerServer;
> if (secureResources != null) {
> tcpPeerServer = new TcpPeerServer(secureResources);
> } else {
> tcpPeerServer = new TcpPeerServer(dnConf.socketWriteTimeout,
> DataNode.getStreamingAddr(conf));
> }
>
> tcpPeerServer.setReceiveBufferSize(HdfsConstants.DEFAULT_DATA_SOCKET_SIZE);
> {code}
> The last line sets SO_RCVBUF explicitly, thus disabling tcp auto-tuning on
> some system.
> Shall we make this behavior configurable?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8762) Erasure Coding: the log of each streamer should show its index
[
https://issues.apache.org/jira/browse/HDFS-8762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang updated HDFS-8762:
Parent Issue: HDFS-8031 (was: HDFS-7285)
> Erasure Coding: the log of each streamer should show its index
> --
>
> Key: HDFS-8762
> URL: https://issues.apache.org/jira/browse/HDFS-8762
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Li Bo
>Assignee: Li Bo
> Attachments: HDFS-8762-HDFS-7285-001.patch,
> HDFS-8762-HDFS-7285-002.patch
>
>
> The log in {{DataStreamer}} doesn't show which streamer it's generated from.
> In order to make log information more convenient for debugging, each log
> should include the index of the streamer it's generated from.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
