[jira] [Updated] (HDFS-8493) Consolidate truncate() related implementation in a single class
[ https://issues.apache.org/jira/browse/HDFS-8493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rakesh R updated HDFS-8493: --- Status: Patch Available (was: Open) > Consolidate truncate() related implementation in a single class > --- > > Key: HDFS-8493 > URL: https://issues.apache.org/jira/browse/HDFS-8493 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Haohui Mai >Assignee: Rakesh R > Attachments: HDFS-8493-001.patch, HDFS-8493-002.patch, > HDFS-8493-003.patch, HDFS-8493-004.patch, HDFS-8493-005.patch, > HDFS-8493-006.patch > > > This jira proposes to consolidate truncate() related methods into a single > class. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8480) Fix performance and timeout issues in HDFS-7929: use hard-links instead of copying edit logs
[
https://issues.apache.org/jira/browse/HDFS-8480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang updated HDFS-8480:
Attachment: HDFS-8480.02.patch
The pre-created
{{hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/name1/current/}}
directory has the following files:
{code}
VERSION
edits_001-031
edits_032-061
fsimage_000
fsimage_000.md5
fsimage_031
fsimage_031.md5
seen_txid
{code}
This emulates a state with last checkpoint at txID 31. During upgrade NN will
try to read and interpret all edit logs after txID 31 (in this case the edit
log file {{edits_032-061}}). If any _new_ edit
logs have an old layout version, an error will occur.
Uploading new patch which only manipulates the layout version of old edit
files. However, the inotify part still fails because of the same
{{EditLogFileInputStream#init()}} error. [~cmccabe] Could you take a look and
see if the test was correctly setup?
> Fix performance and timeout issues in HDFS-7929: use hard-links instead of
> copying edit logs
>
>
> Key: HDFS-8480
> URL: https://issues.apache.org/jira/browse/HDFS-8480
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.0
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
>Priority: Critical
> Attachments: HDFS-8480.00.patch, HDFS-8480.01.patch,
> HDFS-8480.02.patch
>
>
> HDFS-7929 copies existing edit logs to the storage directory of the upgraded
> {{NameNode}}. This slows down the upgrade process. This JIRA aims to use
> hard-linking instead of per-op copying to achieve the same goal.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8580) Erasure coding: Persist cellSize in BlockInfoStriped and StripedBlocksFeature
[
https://issues.apache.org/jira/browse/HDFS-8580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Walter Su updated HDFS-8580:
Attachment: HDFS-8580-HDFS-7285.03.patch
bq. Thus how about just retrieving the cell size from the INode's local striped
block info?
My concern is newly started File has no block. So I add an if-else in Saver.
Uploaded 03 patch.
> Erasure coding: Persist cellSize in BlockInfoStriped and StripedBlocksFeature
> -
>
> Key: HDFS-8580
> URL: https://issues.apache.org/jira/browse/HDFS-8580
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Walter Su
>Assignee: Walter Su
> Attachments: HDFS-8580-HDFS-7285.01.patch,
> HDFS-8580-HDFS-7285.02.patch, HDFS-8580-HDFS-7285.03.patch, HDFS-8580.00.patch
>
>
> Zhe Zhang, Kai Zheng and I had a offline discussion. Here is what we thought:
> Add a cellSize field in BlockInfoStriped as a workaround, and deal with
> memory usage in follow-on.(HDFS-8059)
> discussion in HDFS-8494:
> from Walter Su:
> {quote}
> I think BlockInfoStriped needs to keep cellSize.
> {quote}
> from [~vinayrpet]:
> {quote}
> I too was thinking the same when the FSImageLoader problem has came up. This
> will increase the memory usage by ~4bytes for each block though.
> {quote}
> from [~jingzhao]
> {quote}
> -Also, we should consider adding a chunk size field to StripedBlockProto and
> removing the cell size field from HdfsFileStatus. In this way we can access
> the chunk size information in the storage layer.-
> {quote}
> ==
> update:
> from [~jingzhao]
> {quote}
> For fsimage part, since HDFS-8585 just removes StripedBlockProto, I guess
> what we can do here is to either 1) add the cellSize information into
> StripedBlocksFeature in fsimage.proto, or 2) bring StripedBlockProto back and
> put block info and cell size there.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8592) SafeModeException never get unwrapped
[
https://issues.apache.org/jira/browse/HDFS-8592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14587512#comment-14587512
]
Hudson commented on HDFS-8592:
--
FAILURE: Integrated in Hadoop-trunk-Commit #8024 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/8024/])
HDFS-8592. SafeModeException never get unwrapped. Contributed by Haohui Mai.
(wheat9: rev 32e39d8a29fdea2647b4372d4422246c9521beb7)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SafeModeException.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestSafeMode.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
> SafeModeException never get unwrapped
> -
>
> Key: HDFS-8592
> URL: https://issues.apache.org/jira/browse/HDFS-8592
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Fix For: 2.8.0
>
> Attachments: HDFS-8592.000.patch
>
>
> {{RemoteException#unwrapRemoteException}} fails to instantiate
> {{SafeModeException}} because {{SafeModeException}} does not have the
> corresponding constructor.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8493) Consolidate truncate() related implementation in a single class
[ https://issues.apache.org/jira/browse/HDFS-8493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rakesh R updated HDFS-8493: --- Status: Open (was: Patch Available) > Consolidate truncate() related implementation in a single class > --- > > Key: HDFS-8493 > URL: https://issues.apache.org/jira/browse/HDFS-8493 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Haohui Mai >Assignee: Rakesh R > Attachments: HDFS-8493-001.patch, HDFS-8493-002.patch, > HDFS-8493-003.patch, HDFS-8493-004.patch, HDFS-8493-005.patch, > HDFS-8493-006.patch > > > This jira proposes to consolidate truncate() related methods into a single > class. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HDFS-8609) Dead Code in DFS Util for DFSUtil#substituteForWildcardAddress
[
https://issues.apache.org/jira/browse/HDFS-8609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
surendra singh lilhore reassigned HDFS-8609:
Assignee: surendra singh lilhore
> Dead Code in DFS Util for DFSUtil#substituteForWildcardAddress
> --
>
> Key: HDFS-8609
> URL: https://issues.apache.org/jira/browse/HDFS-8609
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Reporter: Bibin A Chundatt
>Assignee: surendra singh lilhore
>Priority: Minor
>
> Dead code after JDK 1.4
> {code}
> otherHttpAddr = DFSUtil.getInfoServerWithDefaultHost(
> otherIpcAddr.getHostName(), otherNode, scheme).toURL();
> {code}
> In {{DFSUtil#substituteForWildcardAddress}}
> {code}
> if (addr != null && addr.isAnyLocalAddress()) {
> ...
> }
> {code}
> addr.isAnyLocalAddress() will always return false.
> Always the url will be formed with address which is configured in
> hdfs-site.xml .Same will affect bootStrap from NN and ssl certificate check
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-8609) Dead Code in DFS Util for DFSUtil#substituteForWildcardAddress
Bibin A Chundatt created HDFS-8609:
--
Summary: Dead Code in DFS Util for
DFSUtil#substituteForWildcardAddress
Key: HDFS-8609
URL: https://issues.apache.org/jira/browse/HDFS-8609
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Reporter: Bibin A Chundatt
Priority: Minor
Dead code after JDK 1.4
{code}
otherHttpAddr = DFSUtil.getInfoServerWithDefaultHost(
otherIpcAddr.getHostName(), otherNode, scheme).toURL();
{code}
In {{DFSUtil#substituteForWildcardAddress}}
{code}
if (addr != null && addr.isAnyLocalAddress()) {
...
}
{code}
addr.isAnyLocalAddress() will always return false.
Always the url will be formed with address which is configured in
hdfs-site.xml .Same will affect bootStrap from NN and ssl certificate check
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8493) Consolidate truncate() related implementation in a single class
[
https://issues.apache.org/jira/browse/HDFS-8493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14587500#comment-14587500
]
Rakesh R commented on HDFS-8493:
Attached patch fixing few checkstyle comments.
Note: Following checkstyle comments are not related to the patch.
{code}
./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java:325:3:
Method length is 598 lines (max allowed is 150).
./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java:1:
File length is 7,520 lines (max allowed is 2,000).
{code}
> Consolidate truncate() related implementation in a single class
> ---
>
> Key: HDFS-8493
> URL: https://issues.apache.org/jira/browse/HDFS-8493
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Haohui Mai
>Assignee: Rakesh R
> Attachments: HDFS-8493-001.patch, HDFS-8493-002.patch,
> HDFS-8493-003.patch, HDFS-8493-004.patch, HDFS-8493-005.patch,
> HDFS-8493-006.patch
>
>
> This jira proposes to consolidate truncate() related methods into a single
> class.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8493) Consolidate truncate() related implementation in a single class
[ https://issues.apache.org/jira/browse/HDFS-8493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rakesh R updated HDFS-8493: --- Attachment: (was: HDFS-8493-006.patch) > Consolidate truncate() related implementation in a single class > --- > > Key: HDFS-8493 > URL: https://issues.apache.org/jira/browse/HDFS-8493 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Haohui Mai >Assignee: Rakesh R > Attachments: HDFS-8493-001.patch, HDFS-8493-002.patch, > HDFS-8493-003.patch, HDFS-8493-004.patch, HDFS-8493-005.patch, > HDFS-8493-006.patch > > > This jira proposes to consolidate truncate() related methods into a single > class. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8493) Consolidate truncate() related implementation in a single class
[ https://issues.apache.org/jira/browse/HDFS-8493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rakesh R updated HDFS-8493: --- Attachment: HDFS-8493-006.patch > Consolidate truncate() related implementation in a single class > --- > > Key: HDFS-8493 > URL: https://issues.apache.org/jira/browse/HDFS-8493 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Haohui Mai >Assignee: Rakesh R > Attachments: HDFS-8493-001.patch, HDFS-8493-002.patch, > HDFS-8493-003.patch, HDFS-8493-004.patch, HDFS-8493-005.patch, > HDFS-8493-006.patch, HDFS-8493-006.patch > > > This jira proposes to consolidate truncate() related methods into a single > class. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8493) Consolidate truncate() related implementation in a single class
[ https://issues.apache.org/jira/browse/HDFS-8493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rakesh R updated HDFS-8493: --- Attachment: HDFS-8493-006.patch > Consolidate truncate() related implementation in a single class > --- > > Key: HDFS-8493 > URL: https://issues.apache.org/jira/browse/HDFS-8493 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Haohui Mai >Assignee: Rakesh R > Attachments: HDFS-8493-001.patch, HDFS-8493-002.patch, > HDFS-8493-003.patch, HDFS-8493-004.patch, HDFS-8493-005.patch, > HDFS-8493-006.patch > > > This jira proposes to consolidate truncate() related methods into a single > class. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8238) Move ClientProtocol to the hdfs-client
[
https://issues.apache.org/jira/browse/HDFS-8238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14587492#comment-14587492
]
Haohui Mai commented on HDFS-8238:
--
[~tasanuma0829], it looks like there is no need to change the signature of the
constructor {{SafeModeException}} after HDFS-8592 is committed. Can you please
rebase your patch? Thanks.
> Move ClientProtocol to the hdfs-client
> --
>
> Key: HDFS-8238
> URL: https://issues.apache.org/jira/browse/HDFS-8238
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: build
>Reporter: Haohui Mai
>Assignee: Takanobu Asanuma
> Attachments: HDFS-8238.000.patch, HDFS-8238.001.patch,
> HDFS-8238.002.patch
>
>
> The {{ClientProtocol}} class defines the RPC protocol between the NN and the
> client. This jira proposes to move it into the hdfs-client module.
> The jira needs to move:
> * {{o.a.h.hdfs.server.namenode.SafeModeException}} and
> {{o.a.h.hdfs.server.namenode.NotReplicatedYetException}} to the hdfs-client
> package
> * Remove the reference of {{DistributedFileSystem}} in the javadoc
> * Create a copy of {{DFSConfigKeys.DFS_NAMENODE_KERBEROS_PRINCIPAL_KEY}} in
> {{HdfsClientConfigKeys}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8592) SafeModeException never get unwrapped
[
https://issues.apache.org/jira/browse/HDFS-8592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haohui Mai updated HDFS-8592:
-
Resolution: Fixed
Fix Version/s: 2.8.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
I've committed the patch to trunk and branch-2. Thanks Jing for the reviews.
> SafeModeException never get unwrapped
> -
>
> Key: HDFS-8592
> URL: https://issues.apache.org/jira/browse/HDFS-8592
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Fix For: 2.8.0
>
> Attachments: HDFS-8592.000.patch
>
>
> {{RemoteException#unwrapRemoteException}} fails to instantiate
> {{SafeModeException}} because {{SafeModeException}} does not have the
> corresponding constructor.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8592) SafeModeException never get unwrapped
[
https://issues.apache.org/jira/browse/HDFS-8592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14587485#comment-14587485
]
Jing Zhao commented on HDFS-8592:
-
+1
> SafeModeException never get unwrapped
> -
>
> Key: HDFS-8592
> URL: https://issues.apache.org/jira/browse/HDFS-8592
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Attachments: HDFS-8592.000.patch
>
>
> {{RemoteException#unwrapRemoteException}} fails to instantiate
> {{SafeModeException}} because {{SafeModeException}} does not have the
> corresponding constructor.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8544) Incorrect port specified in HFTP Guide document in branch-2
[
https://issues.apache.org/jira/browse/HDFS-8544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14587480#comment-14587480
]
Brahma Reddy Battula commented on HDFS-8544:
Thanks a lot [~yzhangal] for review and commit...
> Incorrect port specified in HFTP Guide document in branch-2
> ---
>
> Key: HDFS-8544
> URL: https://issues.apache.org/jira/browse/HDFS-8544
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: documentation
>Affects Versions: 2.7.0
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Fix For: 2.7.1
>
> Attachments: HDFS-8544.patch
>
>
> IN
> https://hadoop.apache.org/docs/r2.7.0/hadoop-project-dist/hadoop-hdfs/Hftp.html
>
> *Given HTTP port instead of RPC port In following sentence*
> HFTP is primarily useful if you have multiple HDFS clusters with different
> versions and you need to move data from one to another. HFTP is
> wire-compatible even between different versions of HDFS. For example, you can
> do things like: *{color:red}hadoop distcp -i hftp://sourceFS:50070/src
> hdfs://destFS:50070/dest{color}* . Note that HFTP is read-only so the
> destination must be an HDFS filesystem. (Also, in this example, the distcp
> should be run using the configuraton of the new filesystem.)
> *Expected:*
> {color:green}hdfs://destFS:/dest{color}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8544) Incorrect port specified in HFTP Guide document in branch-2
[
https://issues.apache.org/jira/browse/HDFS-8544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongjun Zhang updated HDFS-8544:
Resolution: Fixed
Fix Version/s: 2.7.1
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
> Incorrect port specified in HFTP Guide document in branch-2
> ---
>
> Key: HDFS-8544
> URL: https://issues.apache.org/jira/browse/HDFS-8544
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: documentation
>Affects Versions: 2.7.0
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Fix For: 2.7.1
>
> Attachments: HDFS-8544.patch
>
>
> IN
> https://hadoop.apache.org/docs/r2.7.0/hadoop-project-dist/hadoop-hdfs/Hftp.html
>
> *Given HTTP port instead of RPC port In following sentence*
> HFTP is primarily useful if you have multiple HDFS clusters with different
> versions and you need to move data from one to another. HFTP is
> wire-compatible even between different versions of HDFS. For example, you can
> do things like: *{color:red}hadoop distcp -i hftp://sourceFS:50070/src
> hdfs://destFS:50070/dest{color}* . Note that HFTP is read-only so the
> destination must be an HDFS filesystem. (Also, in this example, the distcp
> should be run using the configuraton of the new filesystem.)
> *Expected:*
> {color:green}hdfs://destFS:/dest{color}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8544) Incorrect port specified in HFTP Guide document in branch-2
[
https://issues.apache.org/jira/browse/HDFS-8544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14587475#comment-14587475
]
Yongjun Zhang commented on HDFS-8544:
-
Thanks Brahma for the contribution, I committed to branch-2 and branch-2.7.
> Incorrect port specified in HFTP Guide document in branch-2
> ---
>
> Key: HDFS-8544
> URL: https://issues.apache.org/jira/browse/HDFS-8544
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: documentation
>Affects Versions: 2.7.0
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Attachments: HDFS-8544.patch
>
>
> IN
> https://hadoop.apache.org/docs/r2.7.0/hadoop-project-dist/hadoop-hdfs/Hftp.html
>
> *Given HTTP port instead of RPC port In following sentence*
> HFTP is primarily useful if you have multiple HDFS clusters with different
> versions and you need to move data from one to another. HFTP is
> wire-compatible even between different versions of HDFS. For example, you can
> do things like: *{color:red}hadoop distcp -i hftp://sourceFS:50070/src
> hdfs://destFS:50070/dest{color}* . Note that HFTP is read-only so the
> destination must be an HDFS filesystem. (Also, in this example, the distcp
> should be run using the configuraton of the new filesystem.)
> *Expected:*
> {color:green}hdfs://destFS:/dest{color}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8597) Fix TestFSImage#testZeroBlockSize on Windows
[
https://issues.apache.org/jira/browse/HDFS-8597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoyu Yao updated HDFS-8597:
-
Attachment: HDFS-8597.01.patch
Update patch that addresses the checkstyle issue by removing the unused import.
> Fix TestFSImage#testZeroBlockSize on Windows
>
>
> Key: HDFS-8597
> URL: https://issues.apache.org/jira/browse/HDFS-8597
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, test
>Affects Versions: 2.6.0
>Reporter: Xiaoyu Yao
>Assignee: Xiaoyu Yao
> Attachments: HDFS-8597.00.patch, HDFS-8597.01.patch
>
>
> The last portion of the dfs.datanode.data.dir is incorrectly formatted.
> {code}2015-06-14 09:44:37,133 INFO hdfs.MiniDFSCluster
> (MiniDFSCluster.java:startDataNodes(1413)) - Starting DataNode 0 with
> dfs.datanode.data.dir:
> file://C:\Users\xiaoyu\hadoop\trunk\hadoop\hadoop-hdfs-project\hadoop-hdfs\target/test/dfs/data
> 2015-06-14 09:44:37,141 ERROR common.Util (Util.java:stringAsURI(50)) -
> Syntax error in URI
> file://C:\Users\xiaoyu\hadoop\trunk\hadoop\hadoop-hdfs-project\hadoop-hdfs\target/test/dfs/data.
> Please check hdfs configuration.
> java.net.URISyntaxException: Illegal character in authority at index 7:
> file://C:\Users\xiaoyu\hadoop\trunk\hadoop\hadoop-hdfs-project\hadoop-hdfs\target/test/dfs/data
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8544) Incorrect port specified in HFTP Guide document in branch-2
[
https://issues.apache.org/jira/browse/HDFS-8544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongjun Zhang updated HDFS-8544:
Summary: Incorrect port specified in HFTP Guide document in branch-2 (was:
[ HFTP ] Wrongly given HTTP port instead of RPC port)
> Incorrect port specified in HFTP Guide document in branch-2
> ---
>
> Key: HDFS-8544
> URL: https://issues.apache.org/jira/browse/HDFS-8544
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: documentation
>Affects Versions: 2.7.0
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Attachments: HDFS-8544.patch
>
>
> IN
> https://hadoop.apache.org/docs/r2.7.0/hadoop-project-dist/hadoop-hdfs/Hftp.html
>
> *Given HTTP port instead of RPC port In following sentence*
> HFTP is primarily useful if you have multiple HDFS clusters with different
> versions and you need to move data from one to another. HFTP is
> wire-compatible even between different versions of HDFS. For example, you can
> do things like: *{color:red}hadoop distcp -i hftp://sourceFS:50070/src
> hdfs://destFS:50070/dest{color}* . Note that HFTP is read-only so the
> destination must be an HDFS filesystem. (Also, in this example, the distcp
> should be run using the configuraton of the new filesystem.)
> *Expected:*
> {color:green}hdfs://destFS:/dest{color}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8544) [ HFTP ] Wrongly given HTTP port instead of RPC port
[
https://issues.apache.org/jira/browse/HDFS-8544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14587463#comment-14587463
]
Yongjun Zhang commented on HDFS-8544:
-
Hi [~brahmareddy], thanks for the good catch and the patch. +1. I will commit
shortly.
> [ HFTP ] Wrongly given HTTP port instead of RPC port
>
>
> Key: HDFS-8544
> URL: https://issues.apache.org/jira/browse/HDFS-8544
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: documentation
>Affects Versions: 2.7.0
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Attachments: HDFS-8544.patch
>
>
> IN
> https://hadoop.apache.org/docs/r2.7.0/hadoop-project-dist/hadoop-hdfs/Hftp.html
>
> *Given HTTP port instead of RPC port In following sentence*
> HFTP is primarily useful if you have multiple HDFS clusters with different
> versions and you need to move data from one to another. HFTP is
> wire-compatible even between different versions of HDFS. For example, you can
> do things like: *{color:red}hadoop distcp -i hftp://sourceFS:50070/src
> hdfs://destFS:50070/dest{color}* . Note that HFTP is read-only so the
> destination must be an HDFS filesystem. (Also, in this example, the distcp
> should be run using the configuraton of the new filesystem.)
> *Expected:*
> {color:green}hdfs://destFS:/dest{color}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8480) Fix performance and timeout issues in HDFS-7929: use hard-links instead of copying edit logs
[
https://issues.apache.org/jira/browse/HDFS-8480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14587427#comment-14587427
]
Zhe Zhang commented on HDFS-8480:
-
Thanks for the comments Vinod and Arpit.
I tried the patch again and got the same failure. To verify that the failure is
caused by edit log version I changed the following line and removed the {{-1}}:
{code}
out.create(NameNodeLayoutVersion.CURRENT_LAYOUT_VERSION - 1); ->
out.create(NameNodeLayoutVersion.CURRENT_LAYOUT_VERSION);
{code}
And then the test passes. I think this expected when
{{EditLogFileInputStream#init()}} is called with {{verifyLayoutVersion}} as
true.
> Fix performance and timeout issues in HDFS-7929: use hard-links instead of
> copying edit logs
>
>
> Key: HDFS-8480
> URL: https://issues.apache.org/jira/browse/HDFS-8480
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.0
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
>Priority: Critical
> Attachments: HDFS-8480.00.patch, HDFS-8480.01.patch
>
>
> HDFS-7929 copies existing edit logs to the storage directory of the upgraded
> {{NameNode}}. This slows down the upgrade process. This JIRA aims to use
> hard-linking instead of per-op copying to achieve the same goal.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8238) Move ClientProtocol to the hdfs-client
[
https://issues.apache.org/jira/browse/HDFS-8238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14587426#comment-14587426
]
Hadoop QA commented on HDFS-8238:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 18m 29s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | javac | 7m 30s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 38s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 48s | The applied patch generated
127 new checkstyle issues (total was 35, now 162). |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 38s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 35s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 4m 6s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 14s | Pre-build of native portion |
| {color:green}+1{color} | hdfs tests | 161m 10s | Tests passed in hadoop-hdfs.
|
| {color:green}+1{color} | hdfs tests | 0m 16s | Tests passed in
hadoop-hdfs-client. |
| | | 208m 50s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12739739/HDFS-8238.002.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 175e6d1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/11364/artifact/patchprocess/diffcheckstylehadoop-hdfs-client.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/11364/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| hadoop-hdfs-client test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/11364/artifact/patchprocess/testrun_hadoop-hdfs-client.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/11364/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/11364/console |
This message was automatically generated.
> Move ClientProtocol to the hdfs-client
> --
>
> Key: HDFS-8238
> URL: https://issues.apache.org/jira/browse/HDFS-8238
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: build
>Reporter: Haohui Mai
>Assignee: Takanobu Asanuma
> Attachments: HDFS-8238.000.patch, HDFS-8238.001.patch,
> HDFS-8238.002.patch
>
>
> The {{ClientProtocol}} class defines the RPC protocol between the NN and the
> client. This jira proposes to move it into the hdfs-client module.
> The jira needs to move:
> * {{o.a.h.hdfs.server.namenode.SafeModeException}} and
> {{o.a.h.hdfs.server.namenode.NotReplicatedYetException}} to the hdfs-client
> package
> * Remove the reference of {{DistributedFileSystem}} in the javadoc
> * Create a copy of {{DFSConfigKeys.DFS_NAMENODE_KERBEROS_PRINCIPAL_KEY}} in
> {{HdfsClientConfigKeys}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8487) Merge BlockInfo-related code changes from HDFS-7285 into trunk and branch-2
[
https://issues.apache.org/jira/browse/HDFS-8487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14587394#comment-14587394
]
Zhe Zhang commented on HDFS-8487:
-
Thanks for the comment Joe. I just updated the JIRA description.
> Merge BlockInfo-related code changes from HDFS-7285 into trunk and branch-2
> ---
>
> Key: HDFS-8487
> URL: https://issues.apache.org/jira/browse/HDFS-8487
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: namenode
>Affects Versions: 2.7.0
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Fix For: 2.8.0
>
>
> Per offline discussion with [~andrew.wang], for easier and cleaner reviewing,
> we should probably shrink the size of the consolidated HDFS-7285 patch by
> merging some mechanical changes that are unrelated to EC-specific logic to
> trunk first. Those include renaming, subclassing, interfaces, and so forth.
> This umbrella JIRA specifically aims to merge code changes around
> {{BlockInfo}} and {{BlockInfoContiguous}} back into trunk.
> The structure of {{BlockInfo}} -related classes are shown below:
> {code}
> BlockInfo (abstract)
>/ \
> BlockInfoStriped BlockInfoContiguous
>||
>| BlockInfoUC |
>| (interface) |
>| / \ |
> BlockInfoStripedUC BlockInfoContiguousUC
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8487) Merge BlockInfo-related code changes from HDFS-7285 into trunk and branch-2
[
https://issues.apache.org/jira/browse/HDFS-8487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang updated HDFS-8487:
Summary: Merge BlockInfo-related code changes from HDFS-7285 into trunk and
branch-2 (was: Merge BlockInfo-related code changes from HDFS-7285 into trunk)
> Merge BlockInfo-related code changes from HDFS-7285 into trunk and branch-2
> ---
>
> Key: HDFS-8487
> URL: https://issues.apache.org/jira/browse/HDFS-8487
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: namenode
>Affects Versions: 2.7.0
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Fix For: 2.8.0
>
>
> Per offline discussion with [~andrew.wang], for easier and cleaner reviewing,
> we should probably shrink the size of the consolidated HDFS-7285 patch by
> merging some mechanical changes that are unrelated to EC-specific logic to
> trunk first. Those include renaming, subclassing, interfaces, and so forth.
> This umbrella JIRA specifically aims to merge code changes around
> {{BlockInfo}} and {{BlockInfoContiguous}} back into trunk.
> The structure of {{BlockInfo}} -related classes are shown below:
> {code}
> BlockInfo (abstract)
>/ \
> BlockInfoStriped BlockInfoContiguous
>||
>| BlockInfoUC |
>| (interface) |
>| / \ |
> BlockInfoStripedUC BlockInfoContiguousUC
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8576) Lease recovery should return true if the lease can be released and the file can be closed
[ https://issues.apache.org/jira/browse/HDFS-8576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14587392#comment-14587392 ] J.Andreina commented on HDFS-8576: -- Thanks [~szetszwo] for the commit. > Lease recovery should return true if the lease can be released and the file > can be closed > -- > > Key: HDFS-8576 > URL: https://issues.apache.org/jira/browse/HDFS-8576 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: J.Andreina >Assignee: J.Andreina > Fix For: 2.7.1 > > Attachments: HDFS-8576.1.patch, HDFS-8576.2.patch > > > FSNamesystem#recoverLease , returns false eventhough lease recover happens. > Hence only on second retry for recovering lease on a file ,returns success > after checking if the file is not underconstruction. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8246) Get HDFS file name based on block pool id and block id
[ https://issues.apache.org/jira/browse/HDFS-8246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14587364#comment-14587364 ] feng xu commented on HDFS-8246: --- For (2) and (3), it will show HDFS file name and snapshots based on block ID, could you explain why it becomes a problem for files having a lot of blocks? For (4), an user can bypass HDFS and access the blocks files directly from local file system, can namenode enforce access control in this case? Instead a local file system extension can be complementary to HDFS security. > Get HDFS file name based on block pool id and block id > -- > > Key: HDFS-8246 > URL: https://issues.apache.org/jira/browse/HDFS-8246 > Project: Hadoop HDFS > Issue Type: New Feature > Components: HDFS, hdfs-client, namenode >Reporter: feng xu >Assignee: feng xu > Labels: BB2015-05-TBR > Attachments: HDFS-8246.0.patch > > > This feature provides HDFS shell command and C/Java API to retrieve HDFS file > name based on block pool id and block id. > 1. The Java API in class DistributedFileSystem > public String getFileName(String poolId, long blockId) throws IOException > 2. The C API in hdfs.c > char* hdfsGetFileName(hdfsFS fs, const char* poolId, int64_t blockId) > 3. The HDFS shell command > hdfs dfs [generic options] -fn > This feature is useful if you have HDFS block file name in local file system > and want to find out the related HDFS file name in HDFS name space > (http://stackoverflow.com/questions/10881449/how-to-find-file-from-blockname-in-hdfs-hadoop). > Each HDFS block file name in local file system contains both block pool id > and block id, for sample HDFS block file name > /hdfs/1/hadoop/hdfs/data/current/BP-97622798-10.3.11.84-1428081035160/current/finalized/subdir0/subdir0/blk_1073741825, > the block pool id is BP-97622798-10.3.11.84-1428081035160 and the block id > is 1073741825. The block pool id is uniquely related to a HDFS name > node/name space, and the block id is uniquely related to a HDFS file within > a HDFS name node/name space, so the combination of block pool id and a block > id is uniquely related a HDFS file name. > The shell command and C/Java API do not map the block pool id to name node, > so it’s user’s responsibility to talk to the correct name node in federation > environment that has multiple name nodes. The block pool id is used by name > node to check if the user is talking with the correct name node. > The implementation is straightforward. The client request to get HDFS file > name reaches the new method String getFileName(String poolId, long blockId) > in FSNamesystem in name node through RPC, and the new method does the > followings, > (1) Validate the block pool id. > (2) Create Block based on the block id. > (3) Get BlockInfoContiguous from Block. > (4) Get BlockCollection from BlockInfoContiguous. > (5) Get file name from BlockCollection. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8246) Get HDFS file name based on block pool id and block id
[ https://issues.apache.org/jira/browse/HDFS-8246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14587323#comment-14587323 ] Tsz Wo Nicholas Sze commented on HDFS-8246: --- (1) and (5) are just empty statements For (2) and (3), we probably do not want ls to show block IDs as Unix/Linux doesn't. Also, it becomes a problem for files having a lot of blocks. For (4), namenode is enforcing access control already. It seems that we don't really have a use case for the proposed API. > Get HDFS file name based on block pool id and block id > -- > > Key: HDFS-8246 > URL: https://issues.apache.org/jira/browse/HDFS-8246 > Project: Hadoop HDFS > Issue Type: New Feature > Components: HDFS, hdfs-client, namenode >Reporter: feng xu >Assignee: feng xu > Labels: BB2015-05-TBR > Attachments: HDFS-8246.0.patch > > > This feature provides HDFS shell command and C/Java API to retrieve HDFS file > name based on block pool id and block id. > 1. The Java API in class DistributedFileSystem > public String getFileName(String poolId, long blockId) throws IOException > 2. The C API in hdfs.c > char* hdfsGetFileName(hdfsFS fs, const char* poolId, int64_t blockId) > 3. The HDFS shell command > hdfs dfs [generic options] -fn > This feature is useful if you have HDFS block file name in local file system > and want to find out the related HDFS file name in HDFS name space > (http://stackoverflow.com/questions/10881449/how-to-find-file-from-blockname-in-hdfs-hadoop). > Each HDFS block file name in local file system contains both block pool id > and block id, for sample HDFS block file name > /hdfs/1/hadoop/hdfs/data/current/BP-97622798-10.3.11.84-1428081035160/current/finalized/subdir0/subdir0/blk_1073741825, > the block pool id is BP-97622798-10.3.11.84-1428081035160 and the block id > is 1073741825. The block pool id is uniquely related to a HDFS name > node/name space, and the block id is uniquely related to a HDFS file within > a HDFS name node/name space, so the combination of block pool id and a block > id is uniquely related a HDFS file name. > The shell command and C/Java API do not map the block pool id to name node, > so it’s user’s responsibility to talk to the correct name node in federation > environment that has multiple name nodes. The block pool id is used by name > node to check if the user is talking with the correct name node. > The implementation is straightforward. The client request to get HDFS file > name reaches the new method String getFileName(String poolId, long blockId) > in FSNamesystem in name node through RPC, and the new method does the > followings, > (1) Validate the block pool id. > (2) Create Block based on the block id. > (3) Get BlockInfoContiguous from Block. > (4) Get BlockCollection from BlockInfoContiguous. > (5) Get file name from BlockCollection. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8574) When block count for a volume exceeds dfs.blockreport.split.threshold, block report causes exception
[
https://issues.apache.org/jira/browse/HDFS-8574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14587324#comment-14587324
]
Ajith S commented on HDFS-8574:
---
Hi [~arpitagarwal]
Thanks for the input. Yes you are right, HDFS was not designed for tiny blocks.
My scenario was like, i wanted to test the NN limits so i inserted 10 million
files with size ~10KB(10KB because i had smaller disk). My DN had one
{{data.dir}} directory, when i faced this exception. But when i increased the
{{data.dir}} to 5, the issue was resolved. Later i checked and came across this
piece of code where the block report was sent per volume of DN. My question is
when we check for overflow, based on number of blocks, then why we split based
on report, as in a single report, there might be still overflow for given limit
{{dfs.blockreport.split.threshold}}
Please correct me if i am wrong
> When block count for a volume exceeds dfs.blockreport.split.threshold, block
> report causes exception
>
>
> Key: HDFS-8574
> URL: https://issues.apache.org/jira/browse/HDFS-8574
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.0
>Reporter: Ajith S
>Assignee: Ajith S
>
> This piece of code in
> {{org.apache.hadoop.hdfs.server.datanode.BPServiceActor.blockReport()}}
> {code}
> // Send one block report per message.
> for (int r = 0; r < reports.length; r++) {
> StorageBlockReport singleReport[] = { reports[r] };
> DatanodeCommand cmd = bpNamenode.blockReport(
> bpRegistration, bpos.getBlockPoolId(), singleReport,
> new BlockReportContext(reports.length, r, reportId));
> numReportsSent++;
> numRPCs++;
> if (cmd != null) {
> cmds.add(cmd);
> }
> {code}
> when a single volume contains many blocks, i.e more than the threshold, it is
> trying to send the entire blockreport in one RPC, causing exception
> {code}
> java.lang.IllegalStateException:
> com.google.protobuf.InvalidProtocolBufferException: Protocol message was too
> large. May be malicious. Use CodedInputStream.setSizeLimit() to increase
> the size limit.
> at
> org.apache.hadoop.hdfs.protocol.BlockListAsLongs$BufferDecoder$1.next(BlockListAsLongs.java:369)
> at
> org.apache.hadoop.hdfs.protocol.BlockListAsLongs$BufferDecoder$1.next(BlockListAsLongs.java:347)
> at
> org.apache.hadoop.hdfs.protocol.BlockListAsLongs$BufferDecoder.getBlockListAsLongs(BlockListAsLongs.java:325)
> at
> org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.blockReport(DatanodeProtocolClientSideTranslatorPB.java:190)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.blockReport(BPServiceActor.java:473)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8246) Get HDFS file name based on block pool id and block id
[ https://issues.apache.org/jira/browse/HDFS-8246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14587313#comment-14587313 ] feng xu commented on HDFS-8246: --- With the enhancement to DistributedFileSystem::listStatus()/hdfsListDirectory(), (1) Any application calling the API will get the functionality without any change. (2) The existing commands like "hdfs dfs –ls" will get the functionality without any change. This is similar to fsck command. (3) The WebHDFS list directory option LISTSTATUS will get the functionality without any change. This is the case that fsck does not cover. (4) Any local file system extension can track the file access in HDFS name space in real time, and make security decision properly with user, process and time information. (5) ……….. > Get HDFS file name based on block pool id and block id > -- > > Key: HDFS-8246 > URL: https://issues.apache.org/jira/browse/HDFS-8246 > Project: Hadoop HDFS > Issue Type: New Feature > Components: HDFS, hdfs-client, namenode >Reporter: feng xu >Assignee: feng xu > Labels: BB2015-05-TBR > Attachments: HDFS-8246.0.patch > > > This feature provides HDFS shell command and C/Java API to retrieve HDFS file > name based on block pool id and block id. > 1. The Java API in class DistributedFileSystem > public String getFileName(String poolId, long blockId) throws IOException > 2. The C API in hdfs.c > char* hdfsGetFileName(hdfsFS fs, const char* poolId, int64_t blockId) > 3. The HDFS shell command > hdfs dfs [generic options] -fn > This feature is useful if you have HDFS block file name in local file system > and want to find out the related HDFS file name in HDFS name space > (http://stackoverflow.com/questions/10881449/how-to-find-file-from-blockname-in-hdfs-hadoop). > Each HDFS block file name in local file system contains both block pool id > and block id, for sample HDFS block file name > /hdfs/1/hadoop/hdfs/data/current/BP-97622798-10.3.11.84-1428081035160/current/finalized/subdir0/subdir0/blk_1073741825, > the block pool id is BP-97622798-10.3.11.84-1428081035160 and the block id > is 1073741825. The block pool id is uniquely related to a HDFS name > node/name space, and the block id is uniquely related to a HDFS file within > a HDFS name node/name space, so the combination of block pool id and a block > id is uniquely related a HDFS file name. > The shell command and C/Java API do not map the block pool id to name node, > so it’s user’s responsibility to talk to the correct name node in federation > environment that has multiple name nodes. The block pool id is used by name > node to check if the user is talking with the correct name node. > The implementation is straightforward. The client request to get HDFS file > name reaches the new method String getFileName(String poolId, long blockId) > in FSNamesystem in name node through RPC, and the new method does the > followings, > (1) Validate the block pool id. > (2) Create Block based on the block id. > (3) Get BlockInfoContiguous from Block. > (4) Get BlockCollection from BlockInfoContiguous. > (5) Get file name from BlockCollection. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8597) Fix TestFSImage#testZeroBlockSize on Windows
[
https://issues.apache.org/jira/browse/HDFS-8597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14587310#comment-14587310
]
Hadoop QA commented on HDFS-8597:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 17m 54s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | javac | 7m 33s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 38s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 2m 16s | The applied patch generated 1
new checkstyle issues (total was 4, now 5). |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 34s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 3m 13s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 15s | Pre-build of native portion |
| {color:green}+1{color} | hdfs tests | 162m 21s | Tests passed in hadoop-hdfs.
|
| | | 208m 44s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12739690/HDFS-8597.00.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 75a2560 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/11363/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/11363/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/11363/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/11363/console |
This message was automatically generated.
> Fix TestFSImage#testZeroBlockSize on Windows
>
>
> Key: HDFS-8597
> URL: https://issues.apache.org/jira/browse/HDFS-8597
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, test
>Affects Versions: 2.6.0
>Reporter: Xiaoyu Yao
>Assignee: Xiaoyu Yao
> Attachments: HDFS-8597.00.patch
>
>
> The last portion of the dfs.datanode.data.dir is incorrectly formatted.
> {code}2015-06-14 09:44:37,133 INFO hdfs.MiniDFSCluster
> (MiniDFSCluster.java:startDataNodes(1413)) - Starting DataNode 0 with
> dfs.datanode.data.dir:
> file://C:\Users\xiaoyu\hadoop\trunk\hadoop\hadoop-hdfs-project\hadoop-hdfs\target/test/dfs/data
> 2015-06-14 09:44:37,141 ERROR common.Util (Util.java:stringAsURI(50)) -
> Syntax error in URI
> file://C:\Users\xiaoyu\hadoop\trunk\hadoop\hadoop-hdfs-project\hadoop-hdfs\target/test/dfs/data.
> Please check hdfs configuration.
> java.net.URISyntaxException: Illegal character in authority at index 7:
> file://C:\Users\xiaoyu\hadoop\trunk\hadoop\hadoop-hdfs-project\hadoop-hdfs\target/test/dfs/data
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8480) Fix performance and timeout issues in HDFS-7929: use hard-links instead of copying edit logs
[
https://issues.apache.org/jira/browse/HDFS-8480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14587279#comment-14587279
]
Arpit Agarwal commented on HDFS-8480:
-
[~zhz] oddly I cannot reproduce the Jenkins test failure with your patch
applied locally.
> Fix performance and timeout issues in HDFS-7929: use hard-links instead of
> copying edit logs
>
>
> Key: HDFS-8480
> URL: https://issues.apache.org/jira/browse/HDFS-8480
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.0
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
>Priority: Critical
> Attachments: HDFS-8480.00.patch, HDFS-8480.01.patch
>
>
> HDFS-7929 copies existing edit logs to the storage directory of the upgraded
> {{NameNode}}. This slows down the upgrade process. This JIRA aims to use
> hard-linking instead of per-op copying to achieve the same goal.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8361) Choose SSD over DISK in block placement
[
https://issues.apache.org/jira/browse/HDFS-8361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsz Wo Nicholas Sze updated HDFS-8361:
--
Resolution: Fixed
Fix Version/s: 2.7.1
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Thanks Joe and Jing for the reviewing.
I have committed this.
> Choose SSD over DISK in block placement
> ---
>
> Key: HDFS-8361
> URL: https://issues.apache.org/jira/browse/HDFS-8361
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Tsz Wo Nicholas Sze
> Fix For: 2.7.1
>
> Attachments: h8361_20150508.patch, h8361_20150612.patch
>
>
> BlockPlacementPolicyDefault chooses the StorageType by iterating the given
> StorageType EnumMap in its natural order (the order in which the enum
> constants are declared). So DISK will be chosen over SSD in One-SSD policy
> since DISK is declared before SSD as shown below. We should choose SSD first.
> {code}
> public enum StorageType {
> DISK(false),
> SSD(false),
> ARCHIVE(false),
> RAM_DISK(true);
> ...
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8483) Erasure coding: test DataNode reporting bad/corrupted blocks which belongs to a striped block.
[ https://issues.apache.org/jira/browse/HDFS-8483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14587246#comment-14587246 ] Takanobu Asanuma commented on HDFS-8483: Thank you for your kind support, Walter! > Erasure coding: test DataNode reporting bad/corrupted blocks which belongs to > a striped block. > -- > > Key: HDFS-8483 > URL: https://issues.apache.org/jira/browse/HDFS-8483 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Takanobu Asanuma >Assignee: Takanobu Asanuma > Fix For: HDFS-7285 > > > We can mimic one/several DataNode(s) reporting bad block(s) (which belong to > a striped block) to the NameNode (through the > DatanodeProtocol#reportBadBlocks call), and check if the > recovery/invalidation work can be correctly scheduled. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8469) Lockfiles are not being created for datanode storage directories
[
https://issues.apache.org/jira/browse/HDFS-8469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14587238#comment-14587238
]
Hadoop QA commented on HDFS-8469:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 18m 19s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | javac | 7m 51s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 45s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 26s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 2m 31s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 2m 0s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 43s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 4m 3s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 4m 14s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 136m 25s | Tests failed in hadoop-hdfs. |
| | | 187m 21s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead |
| | hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes |
| | hadoop.hdfs.server.namenode.TestFileContextAcl |
| | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS |
| | hadoop.hdfs.web.TestWebHDFSXAttr |
| | hadoop.hdfs.server.blockmanagement.TestNodeCount |
| | hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks |
| | hadoop.hdfs.TestDatanodeRegistration |
| | hadoop.hdfs.web.TestWebHDFSAcl |
| | hadoop.hdfs.TestFileCreationDelete |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestDatanodeRestart |
| | hadoop.hdfs.TestSafeMode |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestWriteToReplica |
| | hadoop.hdfs.server.blockmanagement.TestNameNodePrunesMissingStorages |
| | hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration |
| | hadoop.hdfs.TestDecommission |
| | hadoop.hdfs.TestEncryptionZonesWithKMS |
| | hadoop.hdfs.TestSetTimes |
| | hadoop.hdfs.server.datanode.TestBlockScanner |
| | hadoop.hdfs.server.namenode.TestXAttrConfigFlag |
| | hadoop.hdfs.TestRestartDFS |
| | hadoop.hdfs.TestRollingUpgrade |
| | hadoop.hdfs.server.blockmanagement.TestOverReplicatedBlocks |
| | hadoop.hdfs.TestDataTransferKeepalive |
| | hadoop.hdfs.TestRenameWhileOpen |
| | hadoop.hdfs.TestDFSClientExcludedNodes |
| | hadoop.hdfs.TestFileCreation |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.TestDFSShell |
| | hadoop.hdfs.server.namenode.TestProcessCorruptBlocks |
| | hadoop.hdfs.TestDistributedFileSystem |
| | hadoop.hdfs.server.namenode.TestNameNodeAcl |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery
|
| | hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade |
| | hadoop.hdfs.TestLeaseRecovery |
| Timed out tests | org.apache.hadoop.hdfs.TestReplication |
| | org.apache.hadoop.hdfs.TestPread |
| | org.apache.hadoop.hdfs.qjournal.client.TestQJMWithFaults |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12734900/HDFS-8469.001.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 04c9a07 |
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/11362/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/11362/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/11362/console |
This message was automatically generated.
> Lockfiles are not being created for datanode storage directories
>
>
> Key: HDFS-8469
> URL: https://issues.apache.org/jira/browse/HDFS-8469
> Project: Hadoop HDFS
> Issue Type:
[jira] [Commented] (HDFS-8238) Move ClientProtocol to the hdfs-client
[
https://issues.apache.org/jira/browse/HDFS-8238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14587235#comment-14587235
]
Takanobu Asanuma commented on HDFS-8238:
I attached the patch with {{git diff -M}}. Please review it? Thank you.
> Move ClientProtocol to the hdfs-client
> --
>
> Key: HDFS-8238
> URL: https://issues.apache.org/jira/browse/HDFS-8238
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: build
>Reporter: Haohui Mai
>Assignee: Takanobu Asanuma
> Attachments: HDFS-8238.000.patch, HDFS-8238.001.patch,
> HDFS-8238.002.patch
>
>
> The {{ClientProtocol}} class defines the RPC protocol between the NN and the
> client. This jira proposes to move it into the hdfs-client module.
> The jira needs to move:
> * {{o.a.h.hdfs.server.namenode.SafeModeException}} and
> {{o.a.h.hdfs.server.namenode.NotReplicatedYetException}} to the hdfs-client
> package
> * Remove the reference of {{DistributedFileSystem}} in the javadoc
> * Create a copy of {{DFSConfigKeys.DFS_NAMENODE_KERBEROS_PRINCIPAL_KEY}} in
> {{HdfsClientConfigKeys}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8238) Move ClientProtocol to the hdfs-client
[
https://issues.apache.org/jira/browse/HDFS-8238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takanobu Asanuma updated HDFS-8238:
---
Attachment: HDFS-8238.002.patch
> Move ClientProtocol to the hdfs-client
> --
>
> Key: HDFS-8238
> URL: https://issues.apache.org/jira/browse/HDFS-8238
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: build
>Reporter: Haohui Mai
>Assignee: Takanobu Asanuma
> Attachments: HDFS-8238.000.patch, HDFS-8238.001.patch,
> HDFS-8238.002.patch
>
>
> The {{ClientProtocol}} class defines the RPC protocol between the NN and the
> client. This jira proposes to move it into the hdfs-client module.
> The jira needs to move:
> * {{o.a.h.hdfs.server.namenode.SafeModeException}} and
> {{o.a.h.hdfs.server.namenode.NotReplicatedYetException}} to the hdfs-client
> package
> * Remove the reference of {{DistributedFileSystem}} in the javadoc
> * Create a copy of {{DFSConfigKeys.DFS_NAMENODE_KERBEROS_PRINCIPAL_KEY}} in
> {{HdfsClientConfigKeys}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8602) Erasure Coding: Client can't read(decode) the EC files which have corrupt blocks.
[
https://issues.apache.org/jira/browse/HDFS-8602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14587220#comment-14587220
]
Kai Sasaki commented on HDFS-8602:
--
{code}
while (true) {
DNAddrPair addressPair = chooseDataNode(block, null);
try {
actualGetFromOneDataNode(addressPair, block, start, end,
buf, offset, corruptedBlockMap);
return;
} catch (IOException e) {
// Ignore. Already processed inside the function.
// Loop through to try the next node.
}
}
{code}
[~walter.k.su] Thank you for advice! {DFSInputStream#fetchBlockByteRange}
ignores. {{actualGetFromOneDataNode}} seems handling next datanode. These logic
does not implemented in {{DFSStripedInputStream}}. Thank you.
> Erasure Coding: Client can't read(decode) the EC files which have corrupt
> blocks.
> -
>
> Key: HDFS-8602
> URL: https://issues.apache.org/jira/browse/HDFS-8602
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Takanobu Asanuma
>Assignee: Kai Sasaki
> Fix For: HDFS-7285
>
>
> Before the DataNode(s) reporting bad block(s), when Client reads the EC file
> which has bad blocks, Client gets hung up. And there are no error messages.
> (When Client reads the replicated file which has bad blocks, the bad blocks
> are reconstructed at the same time, and Client can reads it.)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8361) Choose SSD over DISK in block placement
[
https://issues.apache.org/jira/browse/HDFS-8361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14587206#comment-14587206
]
Hudson commented on HDFS-8361:
--
FAILURE: Integrated in Hadoop-trunk-Commit #8022 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/8022/])
HDFS-8361. Choose SSD over DISK in block placement. (szetszwo: rev
175e6d120fc34710048c2e9512dd270410172803)
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockStoragePolicy.java
*
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
> Choose SSD over DISK in block placement
> ---
>
> Key: HDFS-8361
> URL: https://issues.apache.org/jira/browse/HDFS-8361
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Tsz Wo Nicholas Sze
> Attachments: h8361_20150508.patch, h8361_20150612.patch
>
>
> BlockPlacementPolicyDefault chooses the StorageType by iterating the given
> StorageType EnumMap in its natural order (the order in which the enum
> constants are declared). So DISK will be chosen over SSD in One-SSD policy
> since DISK is declared before SSD as shown below. We should choose SSD first.
> {code}
> public enum StorageType {
> DISK(false),
> SSD(false),
> ARCHIVE(false),
> RAM_DISK(true);
> ...
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8540) Mover should exit with NO_MOVE_BLOCK if no block can be moved
[
https://issues.apache.org/jira/browse/HDFS-8540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14587193#comment-14587193
]
Hudson commented on HDFS-8540:
--
FAILURE: Integrated in Hadoop-trunk-Commit #8021 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/8021/])
Move HDFS-8540 to 2.8 in CHANGES.txt. (szetszwo: rev
1b6695a4c0d76fe18d6524cc1379bc1185708c6f)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
> Mover should exit with NO_MOVE_BLOCK if no block can be moved
> -
>
> Key: HDFS-8540
> URL: https://issues.apache.org/jira/browse/HDFS-8540
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: balancer & mover
>Reporter: Tsz Wo Nicholas Sze
>Assignee: surendra singh lilhore
> Fix For: 2.8.0
>
> Attachments: HDFS-8540.patch, HDFS-8540_1.patch, HDFS-8540_2.patch,
> HDFS-8540_3.patch
>
>
> When there are files not satisfying their storage policy and no move is
> possible, Mover exits with SUCCESS. It should exit with NO_MOVE_BLOCK.
> The bug seems in the following code. When StorageTypeDiff is not empty and
> scheduleMoves4Block return false, it does not update hasRemaining. Also,
> there is no indication of "No block can be moved" for the entire iteration.
> {code}
> //Mover.processFile(..)
> if (!diff.removeOverlap(true)) {
> if (scheduleMoves4Block(diff, lb, ecSchema)) {
> hasRemaining |= (diff.existing.size() > 1 &&
> diff.expected.size() > 1);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8540) Mover should exit with NO_MOVE_BLOCK if no block can be moved
[
https://issues.apache.org/jira/browse/HDFS-8540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsz Wo Nicholas Sze updated HDFS-8540:
--
Resolution: Fixed
Fix Version/s: 2.8.0
Status: Resolved (was: Patch Available)
I have committed this. Thanks, surendra!
> Mover should exit with NO_MOVE_BLOCK if no block can be moved
> -
>
> Key: HDFS-8540
> URL: https://issues.apache.org/jira/browse/HDFS-8540
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: balancer & mover
>Reporter: Tsz Wo Nicholas Sze
>Assignee: surendra singh lilhore
> Fix For: 2.8.0
>
> Attachments: HDFS-8540.patch, HDFS-8540_1.patch, HDFS-8540_2.patch,
> HDFS-8540_3.patch
>
>
> When there are files not satisfying their storage policy and no move is
> possible, Mover exits with SUCCESS. It should exit with NO_MOVE_BLOCK.
> The bug seems in the following code. When StorageTypeDiff is not empty and
> scheduleMoves4Block return false, it does not update hasRemaining. Also,
> there is no indication of "No block can be moved" for the entire iteration.
> {code}
> //Mover.processFile(..)
> if (!diff.removeOverlap(true)) {
> if (scheduleMoves4Block(diff, lb, ecSchema)) {
> hasRemaining |= (diff.existing.size() > 1 &&
> diff.expected.size() > 1);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8487) Merge BlockInfo-related code changes from HDFS-7285 into trunk
[
https://issues.apache.org/jira/browse/HDFS-8487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14587167#comment-14587167
]
Joe Pallas commented on HDFS-8487:
--
So, um, branch-2 is the trunk?
> Merge BlockInfo-related code changes from HDFS-7285 into trunk
> --
>
> Key: HDFS-8487
> URL: https://issues.apache.org/jira/browse/HDFS-8487
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: namenode
>Affects Versions: 2.7.0
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Fix For: 2.8.0
>
>
> Per offline discussion with [~andrew.wang], for easier and cleaner reviewing,
> we should probably shrink the size of the consolidated HDFS-7285 patch by
> merging some mechanical changes that are unrelated to EC-specific logic to
> trunk first. Those include renaming, subclassing, interfaces, and so forth.
> This umbrella JIRA specifically aims to merge code changes around
> {{BlockInfo}} and {{BlockInfoContiguous}} back into trunk.
> The structure of {{BlockInfo}} -related classes are shown below:
> {code}
> BlockInfo (abstract)
>/ \
> BlockInfoStriped BlockInfoContiguous
>||
>| BlockInfoUC |
>| (interface) |
>| / \ |
> BlockInfoStripedUC BlockInfoContiguousUC
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8608) Merge HDFS-7912 to trunk (track BlockInfo instead of Block in UnderReplicatedBlocks and PendingReplicationBlocks)
[
https://issues.apache.org/jira/browse/HDFS-8608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14587164#comment-14587164
]
Hadoop QA commented on HDFS-8608:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 17m 47s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 10 new or modified test files. |
| {color:green}+1{color} | javac | 7m 32s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 42s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 2m 12s | The applied patch generated 6
new checkstyle issues (total was 722, now 719). |
| {color:red}-1{color} | whitespace | 0m 6s | The patch has 9 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 33s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 35s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 3m 14s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 13s | Pre-build of native portion |
| {color:green}+1{color} | hdfs tests | 162m 13s | Tests passed in hadoop-hdfs.
|
| | | 208m 36s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12739678/HDFS-8608.00.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 32ffda1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/11361/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-HDFS-Build/11361/artifact/patchprocess/whitespace.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/11361/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/11361/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/11361/console |
This message was automatically generated.
> Merge HDFS-7912 to trunk (track BlockInfo instead of Block in
> UnderReplicatedBlocks and PendingReplicationBlocks)
> -
>
> Key: HDFS-8608
> URL: https://issues.apache.org/jira/browse/HDFS-8608
> Project: Hadoop HDFS
> Issue Type: New Feature
>Affects Versions: 2.7.0
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Attachments: HDFS-8608.00.patch
>
>
> This JIRA aims to merges HDFS-7912 into trunk to minimize final patch when
> merging the HDFS-7285 (erasure coding) branch.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7923) The DataNodes should rate-limit their full block reports by asking the NN on heartbeat messages
[ https://issues.apache.org/jira/browse/HDFS-7923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14587163#comment-14587163 ] Sanjay Radia commented on HDFS-7923: Have you considered a pull model (NN pulls) which does not risk starvation? > The DataNodes should rate-limit their full block reports by asking the NN on > heartbeat messages > --- > > Key: HDFS-7923 > URL: https://issues.apache.org/jira/browse/HDFS-7923 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 2.8.0 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Fix For: 2.8.0 > > Attachments: HDFS-7923.000.patch, HDFS-7923.001.patch, > HDFS-7923.002.patch, HDFS-7923.003.patch, HDFS-7923.004.patch, > HDFS-7923.006.patch, HDFS-7923.007.patch > > > The DataNodes should rate-limit their full block reports. They can do this > by first sending a heartbeat message to the NN with an optional boolean set > which requests permission to send a full block report. If the NN responds > with another optional boolean set, the DN will send an FBR... if not, it will > wait until later. This can be done compatibly with optional fields. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8361) Choose SSD over DISK in block placement
[
https://issues.apache.org/jira/browse/HDFS-8361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14587159#comment-14587159
]
Joe Pallas commented on HDFS-8361:
--
Yes, thanks.
> Choose SSD over DISK in block placement
> ---
>
> Key: HDFS-8361
> URL: https://issues.apache.org/jira/browse/HDFS-8361
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Tsz Wo Nicholas Sze
> Attachments: h8361_20150508.patch, h8361_20150612.patch
>
>
> BlockPlacementPolicyDefault chooses the StorageType by iterating the given
> StorageType EnumMap in its natural order (the order in which the enum
> constants are declared). So DISK will be chosen over SSD in One-SSD policy
> since DISK is declared before SSD as shown below. We should choose SSD first.
> {code}
> public enum StorageType {
> DISK(false),
> SSD(false),
> ARCHIVE(false),
> RAM_DISK(true);
> ...
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8540) Mover should exit with NO_MOVE_BLOCK if no block can be moved
[
https://issues.apache.org/jira/browse/HDFS-8540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14587155#comment-14587155
]
Hudson commented on HDFS-8540:
--
FAILURE: Integrated in Hadoop-trunk-Commit #8020 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/8020/])
HDFS-8540. Mover should exit with NO_MOVE_BLOCK if no block can be moved.
Contributed by surendra singh lilhore (szetszwo: rev
321940cf19375febe9660e96d905360cfcc15f5f)
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/mover/TestStorageMover.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/mover/Mover.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/mover/TestMover.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
> Mover should exit with NO_MOVE_BLOCK if no block can be moved
> -
>
> Key: HDFS-8540
> URL: https://issues.apache.org/jira/browse/HDFS-8540
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: balancer & mover
>Reporter: Tsz Wo Nicholas Sze
>Assignee: surendra singh lilhore
> Attachments: HDFS-8540.patch, HDFS-8540_1.patch, HDFS-8540_2.patch,
> HDFS-8540_3.patch
>
>
> When there are files not satisfying their storage policy and no move is
> possible, Mover exits with SUCCESS. It should exit with NO_MOVE_BLOCK.
> The bug seems in the following code. When StorageTypeDiff is not empty and
> scheduleMoves4Block return false, it does not update hasRemaining. Also,
> there is no indication of "No block can be moved" for the entire iteration.
> {code}
> //Mover.processFile(..)
> if (!diff.removeOverlap(true)) {
> if (scheduleMoves4Block(diff, lb, ecSchema)) {
> hasRemaining |= (diff.existing.size() > 1 &&
> diff.expected.size() > 1);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8361) Choose SSD over DISK in block placement
[
https://issues.apache.org/jira/browse/HDFS-8361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14587144#comment-14587144
]
Jing Zhao commented on HDFS-8361:
-
Thanks for working on this, Nicholas! It may be easier to follow if exposing
the storage type selection logic in a more explicit way in
BlockPlacementPolicyDefault. But the current fix also looks good to me. +1.
> Choose SSD over DISK in block placement
> ---
>
> Key: HDFS-8361
> URL: https://issues.apache.org/jira/browse/HDFS-8361
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Tsz Wo Nicholas Sze
> Attachments: h8361_20150508.patch, h8361_20150612.patch
>
>
> BlockPlacementPolicyDefault chooses the StorageType by iterating the given
> StorageType EnumMap in its natural order (the order in which the enum
> constants are declared). So DISK will be chosen over SSD in One-SSD policy
> since DISK is declared before SSD as shown below. We should choose SSD first.
> {code}
> public enum StorageType {
> DISK(false),
> SSD(false),
> ARCHIVE(false),
> RAM_DISK(true);
> ...
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8499) Refactor BlockInfo class hierarchy with static helper class
[
https://issues.apache.org/jira/browse/HDFS-8499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang updated HDFS-8499:
Attachment: HDFS-bistriped.patch
With this patch, the {{BlockInfoStriped}} code will be something like the
attached (rebased from HDFS-7716 patch).
Patches like this obviously don't apply on the HDFS-7285 branch (since it
modifies one of the early patches in the branch). So I think we can either
target them for trunk or create a new branch for the merging purpose. Any
advice would be much appreciated.
> Refactor BlockInfo class hierarchy with static helper class
> ---
>
> Key: HDFS-8499
> URL: https://issues.apache.org/jira/browse/HDFS-8499
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: 2.7.0
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Fix For: 2.8.0
>
> Attachments: HDFS-8499.00.patch, HDFS-8499.01.patch,
> HDFS-8499.02.patch, HDFS-8499.03.patch, HDFS-8499.04.patch,
> HDFS-8499.05.patch, HDFS-8499.06.patch, HDFS-8499.07.patch,
> HDFS-8499.UCFeature.patch, HDFS-bistriped.patch
>
>
> In HDFS-7285 branch, the {{BlockInfoUnderConstruction}} interface provides a
> common abstraction for striped and contiguous UC blocks. This JIRA aims to
> merge it to trunk.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8540) Mover should exit with NO_MOVE_BLOCK if no block can be moved
[
https://issues.apache.org/jira/browse/HDFS-8540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsz Wo Nicholas Sze updated HDFS-8540:
--
Hadoop Flags: Reviewed
+1 the new patch looks good.
> Mover should exit with NO_MOVE_BLOCK if no block can be moved
> -
>
> Key: HDFS-8540
> URL: https://issues.apache.org/jira/browse/HDFS-8540
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: balancer & mover
>Reporter: Tsz Wo Nicholas Sze
>Assignee: surendra singh lilhore
> Attachments: HDFS-8540.patch, HDFS-8540_1.patch, HDFS-8540_2.patch,
> HDFS-8540_3.patch
>
>
> When there are files not satisfying their storage policy and no move is
> possible, Mover exits with SUCCESS. It should exit with NO_MOVE_BLOCK.
> The bug seems in the following code. When StorageTypeDiff is not empty and
> scheduleMoves4Block return false, it does not update hasRemaining. Also,
> there is no indication of "No block can be moved" for the entire iteration.
> {code}
> //Mover.processFile(..)
> if (!diff.removeOverlap(true)) {
> if (scheduleMoves4Block(diff, lb, ecSchema)) {
> hasRemaining |= (diff.existing.size() > 1 &&
> diff.expected.size() > 1);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8576) Lease recovery should return true if the lease can be released and the file can be closed
[ https://issues.apache.org/jira/browse/HDFS-8576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo Nicholas Sze updated HDFS-8576: -- Resolution: Fixed Fix Version/s: 2.7.1 Status: Resolved (was: Patch Available) I have committed this. Thanks, J.Andreina! > Lease recovery should return true if the lease can be released and the file > can be closed > -- > > Key: HDFS-8576 > URL: https://issues.apache.org/jira/browse/HDFS-8576 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: J.Andreina >Assignee: J.Andreina > Fix For: 2.7.1 > > Attachments: HDFS-8576.1.patch, HDFS-8576.2.patch > > > FSNamesystem#recoverLease , returns false eventhough lease recover happens. > Hence only on second retry for recovering lease on a file ,returns success > after checking if the file is not underconstruction. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8576) Lease recovery should return true if the lease can be released and the file can be closed
[ https://issues.apache.org/jira/browse/HDFS-8576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14587126#comment-14587126 ] Hudson commented on HDFS-8576: -- FAILURE: Integrated in Hadoop-trunk-Commit #8019 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/8019/]) HDFS-8576. Lease recovery should return true if the lease can be released and the file can be closed. Contributed by J.Andreina (szetszwo: rev 2cb09e98e392feb5732d0754b539240094edc37a) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestLeaseRecovery.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt > Lease recovery should return true if the lease can be released and the file > can be closed > -- > > Key: HDFS-8576 > URL: https://issues.apache.org/jira/browse/HDFS-8576 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: J.Andreina >Assignee: J.Andreina > Fix For: 2.7.1 > > Attachments: HDFS-8576.1.patch, HDFS-8576.2.patch > > > FSNamesystem#recoverLease , returns false eventhough lease recover happens. > Hence only on second retry for recovering lease on a file ,returns success > after checking if the file is not underconstruction. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8543) Erasure Coding: processOverReplicatedBlock() handles striped block
[
https://issues.apache.org/jira/browse/HDFS-8543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14587106#comment-14587106
]
Jing Zhao commented on HDFS-8543:
-
Thanks for working on this, Walter! So some comments other than Zhe's comments:
# One minor is that we can optimize the following code:
{code}
+// for each duplicated index, delete some replicas until only one left
+for (int targetIndex = duplicated.nextSetBit(0); targetIndex >= 0;
+ targetIndex = duplicated.nextSetBit(targetIndex + 1)) {
+ List candidates = new ArrayList<>();
+ for (DatanodeStorageInfo storage : nonExcess) {
+int index = sblk.getStorageBlockIndex(storage);
+if (index == targetIndex) {
+ candidates.add(storage);
+}
+ }
{code}
{{getStorageBlockIndex}} has to go through the whole object triplet array.
Looks like we can avoid making this call in the inner loop by caching the
mapping of storage and index into a temp map.
# The following code in {{chooseExcessReplicasStriped}} works currently but
looks like the 2nd and 3rd parameters are not semantically consistent with the
parameter {{candidates}}? {{candidates}} is for the internal block while the
former two parameters are about the whole block group. Maybe we can use a new
block object with the internal block's ID and replication factor 1 here.
{code}
DatanodeStorageInfo target = placementPolicy.chooseReplicaToDelete(bc,
storedBlock, groupSize, candidates, empty, excessTypes);
{code}
> Erasure Coding: processOverReplicatedBlock() handles striped block
> --
>
> Key: HDFS-8543
> URL: https://issues.apache.org/jira/browse/HDFS-8543
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Walter Su
>Assignee: Walter Su
> Attachments: HDFS-8543-HDFS-7285.01.patch,
> HDFS-8543-HDFS-7285.02.patch, HDFS-8543-HDFS-7285.03.patch,
> HDFS-8543-HDFS-7285.04.patch, HDFS-8543-HDFS-7285.05.patch,
> HDFS-8543-HDFS-7285.06.patch
>
>
> striped block group could be over replicated when: 1.dead DN comes back. 2.
> Balancer/Mover copies block before deletes it.
> This jira add logic for processOverReplicatedBlock() handling striped block
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7923) The DataNodes should rate-limit their full block reports by asking the NN on heartbeat messages
[ https://issues.apache.org/jira/browse/HDFS-7923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14587090#comment-14587090 ] Suresh Srinivas commented on HDFS-7923: --- I just notice this. [~daryn] and I have talked about improving block report processing efficiency. The fact that it can take close to 300 ms is a concern. This change work around that and I see cases where Datanodes can be starved. [~daryn], any comments? > The DataNodes should rate-limit their full block reports by asking the NN on > heartbeat messages > --- > > Key: HDFS-7923 > URL: https://issues.apache.org/jira/browse/HDFS-7923 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 2.8.0 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Fix For: 2.8.0 > > Attachments: HDFS-7923.000.patch, HDFS-7923.001.patch, > HDFS-7923.002.patch, HDFS-7923.003.patch, HDFS-7923.004.patch, > HDFS-7923.006.patch, HDFS-7923.007.patch > > > The DataNodes should rate-limit their full block reports. They can do this > by first sending a heartbeat message to the NN with an optional boolean set > which requests permission to send a full block report. If the NN responds > with another optional boolean set, the DN will send an FBR... if not, it will > wait until later. This can be done compatibly with optional fields. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8576) Lease recovery should return true if the lease can be released and the file can be closed
[ https://issues.apache.org/jira/browse/HDFS-8576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo Nicholas Sze updated HDFS-8576: -- Component/s: namenode Summary: Lease recovery should return true if the lease can be released and the file can be closed (was: Lease recovery returns false , eventhough recovery happens.) Hadoop Flags: Reviewed +1 the new patch looks good. (revised Summary) > Lease recovery should return true if the lease can be released and the file > can be closed > -- > > Key: HDFS-8576 > URL: https://issues.apache.org/jira/browse/HDFS-8576 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: J.Andreina >Assignee: J.Andreina > Attachments: HDFS-8576.1.patch, HDFS-8576.2.patch > > > FSNamesystem#recoverLease , returns false eventhough lease recover happens. > Hence only on second retry for recovering lease on a file ,returns success > after checking if the file is not underconstruction. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8582) Reduce failure messages when running datanode reconfiguration
[
https://issues.apache.org/jira/browse/HDFS-8582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14587086#comment-14587086
]
Hadoop QA commented on HDFS-8582:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 19m 25s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 2 new or modified test files. |
| {color:green}+1{color} | javac | 7m 27s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 36s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 3m 21s | The applied patch generated 8
new checkstyle issues (total was 392, now 395). |
| {color:green}+1{color} | whitespace | 0m 1s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 34s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 5m 3s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | common tests | 23m 2s | Tests passed in
hadoop-common. |
| {color:green}+1{color} | hdfs tests | 161m 31s | Tests passed in hadoop-hdfs.
|
| | | 232m 0s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12739661/HDFS-8582.002.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 32ffda1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/11360/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
|
| hadoop-common test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/11360/artifact/patchprocess/testrun_hadoop-common.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/11360/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/11360/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/11360/console |
This message was automatically generated.
> Reduce failure messages when running datanode reconfiguration
> -
>
> Key: HDFS-8582
> URL: https://issues.apache.org/jira/browse/HDFS-8582
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: HDFS
>Affects Versions: 2.7.0
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
>Priority: Minor
> Attachments: HDFS-8582.000.patch, HDFS-8582.001.patch,
> HDFS-8582.002.patch
>
>
> When running a DN reconfig to hotswap some drives, it spits out this output:
> {noformat}
> $ hdfs dfsadmin -reconfig datanode localhost:9023 status
> 15/06/09 14:58:10 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> Reconfiguring status for DataNode[localhost:9023]: started at Tue Jun 09
> 14:57:37 PDT 2015 and finished at Tue Jun 09 14:57:56 PDT 2015.
> FAILED: Change property
> rpc.engine.org.apache.hadoop.hdfs.protocolPB.ClientDatanodeProtocolPB
> From: "org.apache.hadoop.ipc.ProtobufRpcEngine"
> To: ""
> Error: Property
> rpc.engine.org.apache.hadoop.hdfs.protocolPB.ClientDatanodeProtocolPB is not
> reconfigurable.
> FAILED: Change property mapreduce.client.genericoptionsparser.used
> From: "true"
> To: ""
> Error: Property mapreduce.client.genericoptionsparser.used is not
> reconfigurable.
> FAILED: Change property rpc.engine.org.apache.hadoop.ipc.ProtocolMetaInfoPB
> From: "org.apache.hadoop.ipc.ProtobufRpcEngine"
> To: ""
> Error: Property rpc.engine.org.apache.hadoop.ipc.ProtocolMetaInfoPB
> is not reconfigurable.
> SUCCESS: Change property dfs.datanode.data.dir
> From: "file:///data/1/user/dfs"
> To: "file:///data/1/user/dfs,file:///data/2/user/dfs"
> FAILED: Change property dfs.datanode.startup
> From: "REGULAR"
> To: ""
> Error: Property dfs.datanode.startup is not reconfigurable.
> FAILED: Change property
> rpc.engine.org.apache.hadoop.hdfs.protocolPB.InterDatanodeProtocolPB
> From: "org.apache.hadoop.ipc.ProtobufRpcEngine"
> To: ""
> Error: Property
> rpc.engine.org.apache.ha
[jira] [Commented] (HDFS-7923) The DataNodes should rate-limit their full block reports by asking the NN on heartbeat messages
[ https://issues.apache.org/jira/browse/HDFS-7923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14587054#comment-14587054 ] Sanjay Radia commented on HDFS-7923: How will you ensure that a particular DN does not get staved? i.e. How do you guarantee that BRs will get through. HDFS depends on periodic BRs for correctness. I recall in discussions with Facebook where they changed their HDFS for incremental BRs but still kept full BRs at a lower frequency just for safety. > The DataNodes should rate-limit their full block reports by asking the NN on > heartbeat messages > --- > > Key: HDFS-7923 > URL: https://issues.apache.org/jira/browse/HDFS-7923 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 2.8.0 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Fix For: 2.8.0 > > Attachments: HDFS-7923.000.patch, HDFS-7923.001.patch, > HDFS-7923.002.patch, HDFS-7923.003.patch, HDFS-7923.004.patch, > HDFS-7923.006.patch, HDFS-7923.007.patch > > > The DataNodes should rate-limit their full block reports. They can do this > by first sending a heartbeat message to the NN with an optional boolean set > which requests permission to send a full block report. If the NN responds > with another optional boolean set, the DN will send an FBR... if not, it will > wait until later. This can be done compatibly with optional fields. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-4660) Duplicated checksum on DN in a recovered pipeline
[ https://issues.apache.org/jira/browse/HDFS-4660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14587017#comment-14587017 ] Vinod Kumar Vavilapalli commented on HDFS-4660: --- [~kihwal], I see you marked this as a blocker for 2.7.1. Assuming you can get hold of someone's review bandwidth to get this done soonish, we are good. Otherwise, also given this is a long standing issue, I recommend we track this instead for 2.7.2. What do you think? > Duplicated checksum on DN in a recovered pipeline > - > > Key: HDFS-4660 > URL: https://issues.apache.org/jira/browse/HDFS-4660 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 3.0.0, 2.0.3-alpha >Reporter: Peng Zhang >Assignee: Kihwal Lee >Priority: Blocker > Attachments: HDFS-4660.patch, HDFS-4660.patch, HDFS-4660.v2.patch > > > pipeline DN1 DN2 DN3 > stop DN2 > pipeline added node DN4 located at 2nd position > DN1 DN4 DN3 > recover RBW > DN4 after recover rbw > 2013-04-01 21:02:31,570 INFO > org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Recover > RBW replica > BP-325305253-10.2.201.14-1364820083462:blk_-9076133543772600337_1004 > 2013-04-01 21:02:31,570 INFO > org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: > Recovering ReplicaBeingWritten, blk_-9076133543772600337_1004, RBW > getNumBytes() = 134144 > getBytesOnDisk() = 134144 > getVisibleLength()= 134144 > end at chunk (134144/512=262) > DN3 after recover rbw > 2013-04-01 21:02:31,575 INFO > org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Recover > RBW replica > BP-325305253-10.2.201.14-1364820083462:blk_-9076133543772600337_10042013-04-01 > 21:02:31,575 INFO > org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: > Recovering ReplicaBeingWritten, blk_-9076133543772600337_1004, RBW > getNumBytes() = 134028 > getBytesOnDisk() = 134028 > getVisibleLength()= 134028 > client send packet after recover pipeline > offset=133632 len=1008 > DN4 after flush > 2013-04-01 21:02:31,779 DEBUG > org.apache.hadoop.hdfs.server.datanode.DataNode: FlushOrsync, file > offset:134640; meta offset:1063 > // meta end position should be floor(134640/512)*4 + 7 == 1059, but now it is > 1063. > DN3 after flush > 2013-04-01 21:02:31,782 DEBUG > org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: > BP-325305253-10.2.201.14-1364820083462:blk_-9076133543772600337_1005, > type=LAST_IN_PIPELINE, downstreams=0:[]: enqueue Packet(seqno=219, > lastPacketInBlock=false, offsetInBlock=134640, > ackEnqueueNanoTime=8817026136871545) > 2013-04-01 21:02:31,782 DEBUG > org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Changing > meta file offset of block > BP-325305253-10.2.201.14-1364820083462:blk_-9076133543772600337_1005 from > 1055 to 1051 > 2013-04-01 21:02:31,782 DEBUG > org.apache.hadoop.hdfs.server.datanode.DataNode: FlushOrsync, file > offset:134640; meta offset:1059 > After checking meta on DN4, I found checksum of chunk 262 is duplicated, but > data not. > Later after block was finalized, DN4's scanner detected bad block, and then > reported it to NM. NM send a command to delete this block, and replicate this > block from other DN in pipeline to satisfy duplication num. > I think this is because in BlockReceiver it skips data bytes already written, > but not skips checksum bytes already written. And function > adjustCrcFilePosition is only used for last non-completed chunk, but > not for this situation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8480) Fix performance and timeout issues in HDFS-7929: use hard-links instead of copying edit logs
[
https://issues.apache.org/jira/browse/HDFS-8480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14586919#comment-14586919
]
Vinod Kumar Vavilapalli commented on HDFS-8480:
---
Appreciate the movement on this [~zhz] and [~cmccabe]!
[~zhz], assuming everything else is addressed, can you look at the test issue
reported by Jenkins so that we can get this in 2.7.1? Thanks again!
> Fix performance and timeout issues in HDFS-7929: use hard-links instead of
> copying edit logs
>
>
> Key: HDFS-8480
> URL: https://issues.apache.org/jira/browse/HDFS-8480
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.0
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
>Priority: Critical
> Attachments: HDFS-8480.00.patch, HDFS-8480.01.patch
>
>
> HDFS-7929 copies existing edit logs to the storage directory of the upgraded
> {{NameNode}}. This slows down the upgrade process. This JIRA aims to use
> hard-linking instead of per-op copying to achieve the same goal.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8498) Blocks can be committed with wrong size
[ https://issues.apache.org/jira/browse/HDFS-8498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated HDFS-8498: -- Target Version/s: 2.7.2 (was: 2.7.1) Sounds like a critical issue to me, but got no response. Moving this to 2.7.2. > Blocks can be committed with wrong size > --- > > Key: HDFS-8498 > URL: https://issues.apache.org/jira/browse/HDFS-8498 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.5.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp >Priority: Critical > > When an IBR for a UC block arrives, the NN updates the expected location's > block and replica state _only_ if it's on an unexpected storage for an > expected DN. If it's for an expected storage, only the genstamp is updated. > When the block is committed, and the expected locations are verified, only > the genstamp is checked. The size is not checked but it wasn't updated in > the expected locations anyway. > A faulty client may misreport the size when committing the block. The block > is effectively corrupted. If the NN issues replications, the received IBR is > considered corrupt, the NN invalidates the block, immediately issues another > replication. The NN eventually realizes all the original replicas are > corrupt after full BRs are received from the original DNs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-4660) Duplicated checksum on DN in a recovered pipeline
[ https://issues.apache.org/jira/browse/HDFS-4660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-4660: - Priority: Blocker (was: Critical) Target Version/s: 2.7.1 > Duplicated checksum on DN in a recovered pipeline > - > > Key: HDFS-4660 > URL: https://issues.apache.org/jira/browse/HDFS-4660 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 3.0.0, 2.0.3-alpha >Reporter: Peng Zhang >Assignee: Kihwal Lee >Priority: Blocker > Attachments: HDFS-4660.patch, HDFS-4660.patch, HDFS-4660.v2.patch > > > pipeline DN1 DN2 DN3 > stop DN2 > pipeline added node DN4 located at 2nd position > DN1 DN4 DN3 > recover RBW > DN4 after recover rbw > 2013-04-01 21:02:31,570 INFO > org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Recover > RBW replica > BP-325305253-10.2.201.14-1364820083462:blk_-9076133543772600337_1004 > 2013-04-01 21:02:31,570 INFO > org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: > Recovering ReplicaBeingWritten, blk_-9076133543772600337_1004, RBW > getNumBytes() = 134144 > getBytesOnDisk() = 134144 > getVisibleLength()= 134144 > end at chunk (134144/512=262) > DN3 after recover rbw > 2013-04-01 21:02:31,575 INFO > org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Recover > RBW replica > BP-325305253-10.2.201.14-1364820083462:blk_-9076133543772600337_10042013-04-01 > 21:02:31,575 INFO > org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: > Recovering ReplicaBeingWritten, blk_-9076133543772600337_1004, RBW > getNumBytes() = 134028 > getBytesOnDisk() = 134028 > getVisibleLength()= 134028 > client send packet after recover pipeline > offset=133632 len=1008 > DN4 after flush > 2013-04-01 21:02:31,779 DEBUG > org.apache.hadoop.hdfs.server.datanode.DataNode: FlushOrsync, file > offset:134640; meta offset:1063 > // meta end position should be floor(134640/512)*4 + 7 == 1059, but now it is > 1063. > DN3 after flush > 2013-04-01 21:02:31,782 DEBUG > org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: > BP-325305253-10.2.201.14-1364820083462:blk_-9076133543772600337_1005, > type=LAST_IN_PIPELINE, downstreams=0:[]: enqueue Packet(seqno=219, > lastPacketInBlock=false, offsetInBlock=134640, > ackEnqueueNanoTime=8817026136871545) > 2013-04-01 21:02:31,782 DEBUG > org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Changing > meta file offset of block > BP-325305253-10.2.201.14-1364820083462:blk_-9076133543772600337_1005 from > 1055 to 1051 > 2013-04-01 21:02:31,782 DEBUG > org.apache.hadoop.hdfs.server.datanode.DataNode: FlushOrsync, file > offset:134640; meta offset:1059 > After checking meta on DN4, I found checksum of chunk 262 is duplicated, but > data not. > Later after block was finalized, DN4's scanner detected bad block, and then > reported it to NM. NM send a command to delete this block, and replicate this > block from other DN in pipeline to satisfy duplication num. > I think this is because in BlockReceiver it skips data bytes already written, > but not skips checksum bytes already written. And function > adjustCrcFilePosition is only used for last non-completed chunk, but > not for this situation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8348) WebHDFS Concat - Support sources list passed a POST body
[ https://issues.apache.org/jira/browse/HDFS-8348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-8348: -- Assignee: vishwajeet dusane > WebHDFS Concat - Support sources list passed a POST body > > > Key: HDFS-8348 > URL: https://issues.apache.org/jira/browse/HDFS-8348 > Project: Hadoop HDFS > Issue Type: Improvement > Components: webhdfs >Reporter: vishwajeet dusane >Assignee: vishwajeet dusane > > Problem - Current webhdfs behavior for concat support sources list to be > passed as query parameter. This approach limits on the number of sources list > send as query parameter. > Proposed Solution - Add support to send sources list as part of the request > body instead of query parameter. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8597) Fix TestFSImage#testZeroBlockSize on Windows
[
https://issues.apache.org/jira/browse/HDFS-8597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14586705#comment-14586705
]
Xiaoyu Yao commented on HDFS-8597:
--
The test takes dfs.datanode.data.dir from "hdfs-default.xml"
{code}
dfs.datanode.data.dir
file://${hadoop.tmp.dir}/dfs/data
{code}
The location ends up as
file://C:\Users\xiaoyu\hadoop\trunk\hadoop\hadoop-hdfs-project\hadoop-hdfs\target/test/dfs/data
before passing to Util.stringAsURI(). However, the Java URI class cannot
handle mix '/' and '\' separator in the path, which causes the exception.
The fix is to use Java Path class to normalize the path separator and then
convert it to URI.
> Fix TestFSImage#testZeroBlockSize on Windows
>
>
> Key: HDFS-8597
> URL: https://issues.apache.org/jira/browse/HDFS-8597
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, test
>Affects Versions: 2.6.0
>Reporter: Xiaoyu Yao
>Assignee: Xiaoyu Yao
> Attachments: HDFS-8597.00.patch
>
>
> The last portion of the dfs.datanode.data.dir is incorrectly formatted.
> {code}2015-06-14 09:44:37,133 INFO hdfs.MiniDFSCluster
> (MiniDFSCluster.java:startDataNodes(1413)) - Starting DataNode 0 with
> dfs.datanode.data.dir:
> file://C:\Users\xiaoyu\hadoop\trunk\hadoop\hadoop-hdfs-project\hadoop-hdfs\target/test/dfs/data
> 2015-06-14 09:44:37,141 ERROR common.Util (Util.java:stringAsURI(50)) -
> Syntax error in URI
> file://C:\Users\xiaoyu\hadoop\trunk\hadoop\hadoop-hdfs-project\hadoop-hdfs\target/test/dfs/data.
> Please check hdfs configuration.
> java.net.URISyntaxException: Illegal character in authority at index 7:
> file://C:\Users\xiaoyu\hadoop\trunk\hadoop\hadoop-hdfs-project\hadoop-hdfs\target/test/dfs/data
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8597) Fix TestFSImage#testZeroBlockSize on Windows
[
https://issues.apache.org/jira/browse/HDFS-8597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoyu Yao updated HDFS-8597:
-
Affects Version/s: 2.6.0
> Fix TestFSImage#testZeroBlockSize on Windows
>
>
> Key: HDFS-8597
> URL: https://issues.apache.org/jira/browse/HDFS-8597
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, test
>Affects Versions: 2.6.0
>Reporter: Xiaoyu Yao
>Assignee: Xiaoyu Yao
> Attachments: HDFS-8597.00.patch
>
>
> The last portion of the dfs.datanode.data.dir is incorrectly formatted.
> {code}2015-06-14 09:44:37,133 INFO hdfs.MiniDFSCluster
> (MiniDFSCluster.java:startDataNodes(1413)) - Starting DataNode 0 with
> dfs.datanode.data.dir:
> file://C:\Users\xiaoyu\hadoop\trunk\hadoop\hadoop-hdfs-project\hadoop-hdfs\target/test/dfs/data
> 2015-06-14 09:44:37,141 ERROR common.Util (Util.java:stringAsURI(50)) -
> Syntax error in URI
> file://C:\Users\xiaoyu\hadoop\trunk\hadoop\hadoop-hdfs-project\hadoop-hdfs\target/test/dfs/data.
> Please check hdfs configuration.
> java.net.URISyntaxException: Illegal character in authority at index 7:
> file://C:\Users\xiaoyu\hadoop\trunk\hadoop\hadoop-hdfs-project\hadoop-hdfs\target/test/dfs/data
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8597) Fix TestFSImage#testZeroBlockSize on Windows
[
https://issues.apache.org/jira/browse/HDFS-8597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoyu Yao updated HDFS-8597:
-
Component/s: test
datanode
> Fix TestFSImage#testZeroBlockSize on Windows
>
>
> Key: HDFS-8597
> URL: https://issues.apache.org/jira/browse/HDFS-8597
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, test
>Affects Versions: 2.6.0
>Reporter: Xiaoyu Yao
>Assignee: Xiaoyu Yao
> Attachments: HDFS-8597.00.patch
>
>
> The last portion of the dfs.datanode.data.dir is incorrectly formatted.
> {code}2015-06-14 09:44:37,133 INFO hdfs.MiniDFSCluster
> (MiniDFSCluster.java:startDataNodes(1413)) - Starting DataNode 0 with
> dfs.datanode.data.dir:
> file://C:\Users\xiaoyu\hadoop\trunk\hadoop\hadoop-hdfs-project\hadoop-hdfs\target/test/dfs/data
> 2015-06-14 09:44:37,141 ERROR common.Util (Util.java:stringAsURI(50)) -
> Syntax error in URI
> file://C:\Users\xiaoyu\hadoop\trunk\hadoop\hadoop-hdfs-project\hadoop-hdfs\target/test/dfs/data.
> Please check hdfs configuration.
> java.net.URISyntaxException: Illegal character in authority at index 7:
> file://C:\Users\xiaoyu\hadoop\trunk\hadoop\hadoop-hdfs-project\hadoop-hdfs\target/test/dfs/data
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8597) Fix TestFSImage#testZeroBlockSize on Windows
[
https://issues.apache.org/jira/browse/HDFS-8597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoyu Yao updated HDFS-8597:
-
Status: Patch Available (was: Open)
> Fix TestFSImage#testZeroBlockSize on Windows
>
>
> Key: HDFS-8597
> URL: https://issues.apache.org/jira/browse/HDFS-8597
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, test
>Reporter: Xiaoyu Yao
>Assignee: Xiaoyu Yao
> Attachments: HDFS-8597.00.patch
>
>
> The last portion of the dfs.datanode.data.dir is incorrectly formatted.
> {code}2015-06-14 09:44:37,133 INFO hdfs.MiniDFSCluster
> (MiniDFSCluster.java:startDataNodes(1413)) - Starting DataNode 0 with
> dfs.datanode.data.dir:
> file://C:\Users\xiaoyu\hadoop\trunk\hadoop\hadoop-hdfs-project\hadoop-hdfs\target/test/dfs/data
> 2015-06-14 09:44:37,141 ERROR common.Util (Util.java:stringAsURI(50)) -
> Syntax error in URI
> file://C:\Users\xiaoyu\hadoop\trunk\hadoop\hadoop-hdfs-project\hadoop-hdfs\target/test/dfs/data.
> Please check hdfs configuration.
> java.net.URISyntaxException: Illegal character in authority at index 7:
> file://C:\Users\xiaoyu\hadoop\trunk\hadoop\hadoop-hdfs-project\hadoop-hdfs\target/test/dfs/data
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8597) Fix TestFSImage#testZeroBlockSize on Windows
[
https://issues.apache.org/jira/browse/HDFS-8597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoyu Yao updated HDFS-8597:
-
Attachment: HDFS-8597.00.patch
Attach a patch that fixes the datanode storage location parsing for windows.
> Fix TestFSImage#testZeroBlockSize on Windows
>
>
> Key: HDFS-8597
> URL: https://issues.apache.org/jira/browse/HDFS-8597
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Xiaoyu Yao
>Assignee: Xiaoyu Yao
> Attachments: HDFS-8597.00.patch
>
>
> The last portion of the dfs.datanode.data.dir is incorrectly formatted.
> {code}2015-06-14 09:44:37,133 INFO hdfs.MiniDFSCluster
> (MiniDFSCluster.java:startDataNodes(1413)) - Starting DataNode 0 with
> dfs.datanode.data.dir:
> file://C:\Users\xiaoyu\hadoop\trunk\hadoop\hadoop-hdfs-project\hadoop-hdfs\target/test/dfs/data
> 2015-06-14 09:44:37,141 ERROR common.Util (Util.java:stringAsURI(50)) -
> Syntax error in URI
> file://C:\Users\xiaoyu\hadoop\trunk\hadoop\hadoop-hdfs-project\hadoop-hdfs\target/test/dfs/data.
> Please check hdfs configuration.
> java.net.URISyntaxException: Illegal character in authority at index 7:
> file://C:\Users\xiaoyu\hadoop\trunk\hadoop\hadoop-hdfs-project\hadoop-hdfs\target/test/dfs/data
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8246) Get HDFS file name based on block pool id and block id
[ https://issues.apache.org/jira/browse/HDFS-8246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14586649#comment-14586649 ] Tsz Wo Nicholas Sze commented on HDFS-8246: --- What are the more use cases? > Get HDFS file name based on block pool id and block id > -- > > Key: HDFS-8246 > URL: https://issues.apache.org/jira/browse/HDFS-8246 > Project: Hadoop HDFS > Issue Type: New Feature > Components: HDFS, hdfs-client, namenode >Reporter: feng xu >Assignee: feng xu > Labels: BB2015-05-TBR > Attachments: HDFS-8246.0.patch > > > This feature provides HDFS shell command and C/Java API to retrieve HDFS file > name based on block pool id and block id. > 1. The Java API in class DistributedFileSystem > public String getFileName(String poolId, long blockId) throws IOException > 2. The C API in hdfs.c > char* hdfsGetFileName(hdfsFS fs, const char* poolId, int64_t blockId) > 3. The HDFS shell command > hdfs dfs [generic options] -fn > This feature is useful if you have HDFS block file name in local file system > and want to find out the related HDFS file name in HDFS name space > (http://stackoverflow.com/questions/10881449/how-to-find-file-from-blockname-in-hdfs-hadoop). > Each HDFS block file name in local file system contains both block pool id > and block id, for sample HDFS block file name > /hdfs/1/hadoop/hdfs/data/current/BP-97622798-10.3.11.84-1428081035160/current/finalized/subdir0/subdir0/blk_1073741825, > the block pool id is BP-97622798-10.3.11.84-1428081035160 and the block id > is 1073741825. The block pool id is uniquely related to a HDFS name > node/name space, and the block id is uniquely related to a HDFS file within > a HDFS name node/name space, so the combination of block pool id and a block > id is uniquely related a HDFS file name. > The shell command and C/Java API do not map the block pool id to name node, > so it’s user’s responsibility to talk to the correct name node in federation > environment that has multiple name nodes. The block pool id is used by name > node to check if the user is talking with the correct name node. > The implementation is straightforward. The client request to get HDFS file > name reaches the new method String getFileName(String poolId, long blockId) > in FSNamesystem in name node through RPC, and the new method does the > followings, > (1) Validate the block pool id. > (2) Create Block based on the block id. > (3) Get BlockInfoContiguous from Block. > (4) Get BlockCollection from BlockInfoContiguous. > (5) Get file name from BlockCollection. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8277) Safemode enter fails when Standby NameNode is down
[
https://issues.apache.org/jira/browse/HDFS-8277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14586585#comment-14586585
]
Arpit Agarwal commented on HDFS-8277:
-
bq. NN stays in safemode until one more manual request is made to come out, or
it gets restarted. So there is no need of edit logging for this. Right?
Hi [~vinayrpet] unfortunately this is is a broken especially with HA. If the
command succeeds on just one NN the state is unpredictable. E.g. if SBN is
temporarily unavailable and then the primary NN crashes then we will no longer
be in safe-mode after failover.
bq. But this going to break the compatibility, by breaking the behaviour of
keeping the NN in safemode even after restart.
Agreed that it will be incompatible. However the halfway fix to issue the
command to just the active is worse. Perhaps we don't change anything for now.
> Safemode enter fails when Standby NameNode is down
> --
>
> Key: HDFS-8277
> URL: https://issues.apache.org/jira/browse/HDFS-8277
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha, HDFS, namenode
>Affects Versions: 2.6.0
> Environment: HDP 2.2.0
>Reporter: Hari Sekhon
>Assignee: surendra singh lilhore
>Priority: Minor
> Attachments: HDFS-8277-safemode-edits.patch, HDFS-8277.patch,
> HDFS-8277_1.patch, HDFS-8277_2.patch, HDFS-8277_3.patch, HDFS-8277_4.patch
>
>
> HDFS fails to enter safemode when the Standby NameNode is down (eg. due to
> AMBARI-10536).
> {code}hdfs dfsadmin -safemode enter
> safemode: Call From nn2/x.x.x.x to nn1:8020 failed on connection exception:
> java.net.ConnectException: Connection refused; For more details see:
> http://wiki.apache.org/hadoop/ConnectionRefused{code}
> This appears to be a bug in that it's not trying both NameNodes like the
> standard hdfs client code does, and is instead stopping after getting a
> connection refused from nn1 which is down. I verified normal hadoop fs writes
> and reads via cli did work at this time, using nn2. I happened to run this
> command as the hdfs user on nn2 which was the surviving Active NameNode.
> After I re-bootstrapped the Standby NN to fix it the command worked as
> expected again.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8608) Merge HDFS-7912 to trunk (track BlockInfo instead of Block in UnderReplicatedBlocks and PendingReplicationBlocks)
[ https://issues.apache.org/jira/browse/HDFS-8608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhe Zhang updated HDFS-8608: Attachment: HDFS-8608.00.patch I rebased HDFS-7912 patch and manually resolved some conflicts. Attaching an initial patch to trigger Jenkins. > Merge HDFS-7912 to trunk (track BlockInfo instead of Block in > UnderReplicatedBlocks and PendingReplicationBlocks) > - > > Key: HDFS-8608 > URL: https://issues.apache.org/jira/browse/HDFS-8608 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 2.7.0 >Reporter: Zhe Zhang >Assignee: Zhe Zhang > Attachments: HDFS-8608.00.patch > > > This JIRA aims to merges HDFS-7912 into trunk to minimize final patch when > merging the HDFS-7285 (erasure coding) branch. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-8608) Merge HDFS-7912 to trunk (track BlockInfo instead of Block in UnderReplicatedBlocks and PendingReplicationBlocks)
Zhe Zhang created HDFS-8608: --- Summary: Merge HDFS-7912 to trunk (track BlockInfo instead of Block in UnderReplicatedBlocks and PendingReplicationBlocks) Key: HDFS-8608 URL: https://issues.apache.org/jira/browse/HDFS-8608 Project: Hadoop HDFS Issue Type: New Feature Affects Versions: 2.7.0 Reporter: Zhe Zhang Assignee: Zhe Zhang This JIRA aims to merges HDFS-7912 into trunk to minimize final patch when merging the HDFS-7285 (erasure coding) branch. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7912) Erasure Coding: track BlockInfo instead of Block in UnderReplicatedBlocks and PendingReplicationBlocks
[
https://issues.apache.org/jira/browse/HDFS-7912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14586577#comment-14586577
]
Zhe Zhang commented on HDFS-7912:
-
Thanks for confirming Jing. I just filed HDFS-8608.
> Erasure Coding: track BlockInfo instead of Block in UnderReplicatedBlocks and
> PendingReplicationBlocks
> --
>
> Key: HDFS-7912
> URL: https://issues.apache.org/jira/browse/HDFS-7912
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Jing Zhao
>Assignee: Jing Zhao
> Fix For: HDFS-7285
>
> Attachments: HDFS-7912.000.patch
>
>
> Now with striped blocks and the design that uses a single BlockInfoStriped
> object to track all the corresponding blocks, we need to clearly distinguish
> the type Block and BlockInfo in BlockManager. Specifically, data structures
> like {{UnderReplicatedBlocks}} and {{PendingReplicationBlocks}} should track
> BlockInfo instead of Block in order to support striped block recovery.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8608) Merge HDFS-7912 to trunk (track BlockInfo instead of Block in UnderReplicatedBlocks and PendingReplicationBlocks)
[ https://issues.apache.org/jira/browse/HDFS-8608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhe Zhang updated HDFS-8608: Status: Patch Available (was: Open) > Merge HDFS-7912 to trunk (track BlockInfo instead of Block in > UnderReplicatedBlocks and PendingReplicationBlocks) > - > > Key: HDFS-8608 > URL: https://issues.apache.org/jira/browse/HDFS-8608 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 2.7.0 >Reporter: Zhe Zhang >Assignee: Zhe Zhang > > This JIRA aims to merges HDFS-7912 into trunk to minimize final patch when > merging the HDFS-7285 (erasure coding) branch. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8578) On upgrade, Datanode should process all storage/data dirs in parallel
[
https://issues.apache.org/jira/browse/HDFS-8578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14586471#comment-14586471
]
Colin Patrick McCabe commented on HDFS-8578:
Hi [~vinayrpet], [~raju.bairishetti], [~amareshwari].
I think it's a great idea to do the upgrade of each storage directory in
parallel. Although these upgrades are usually quick, sometimes they aren't.
For example, if there is a slow disk, we don't want to slow down the whole
process. Another reason is that when upgrades are slow, it's almost always
because we are I/O-bound. So it just makes sense to do all the directories
(i.e. hard drives) in parallel.
There are a few cases where we will need to change certain log messages to
include the storage directory path, to avoid confusion when doing things in
parallel. Keep in mind the log messages will appear in parallel, so we won't
be able to rely on the log message ordering to tell us which storage directory
the message pertains to.
{code}
private StorageDirectory loadStorageDirectory(DataNode datanode,
NamespaceInfo nsInfo, File dataDir, StartupOption startOpt) throws
IOException {
...
LOG.info("Formatting ...");
{code}
The "Formatting..." log message must include the directory being formatted.
{code}
private void linkAllBlocks(DataNode datanode, File fromDir, File toDir)
throws IOException {
...
LOG.info( hardLink.linkStats.report() );
}
{code}
Here is another case where the existing LOG is not enough to tell us which
storage directory is being processed.
{code}
245 try {
246 IOException ioe = ioExceptionFuture.get();
...
259 } catch (InterruptedException e) {
260 LOG.error("InterruptedExeption while analyzing" + " blockpool "
261 + nsInfo.getBlockPoolID());
262 }
{code}
If the thread gets an {{InterruptedException}} while waiting for a {{Future}},
you are simply logging a message and giving up on waiting for that {{Future}}.
That's not right. I think this would be easier to get right by using Guava's
{{Uninterruptibles#getUninterruptibly}}. You also should handle
{{CancellationException}}.
Thanks, guys.
> On upgrade, Datanode should process all storage/data dirs in parallel
> -
>
> Key: HDFS-8578
> URL: https://issues.apache.org/jira/browse/HDFS-8578
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Reporter: Raju Bairishetti
>Priority: Critical
> Attachments: HDFS-8578-01.patch
>
>
> Right now, during upgrades datanode is processing all the storage dirs
> sequentially. Assume it takes ~20 mins to process a single storage dir then
> datanode which has ~10 disks will take around 3hours to come up.
> *BlockPoolSliceStorage.java*
> {code}
>for (int idx = 0; idx < getNumStorageDirs(); idx++) {
> doTransition(datanode, getStorageDir(idx), nsInfo, startOpt);
> assert getCTime() == nsInfo.getCTime()
> : "Data-node and name-node CTimes must be the same.";
> }
> {code}
> It would save lots of time during major upgrades if datanode process all
> storagedirs/disks parallelly.
> Can we make datanode to process all storage dirs parallelly?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8582) Reduce failure messages when running datanode reconfiguration
[
https://issues.apache.org/jira/browse/HDFS-8582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14586440#comment-14586440
]
Colin Patrick McCabe commented on HDFS-8582:
I think it's a good idea to have a list of reconfigurable properties
available... the new listAllowed command is nice. Rather than calling it
"listAllowed" (which suggests a permission issue), why not just call it
"properties"? The first part of the command is {{dfs dfsadmin -reconfig
}} so it will be obvious that the properties
pertain to reconfiguration.
{{GetReconfigurationAllowedPropertiesResponse}}: should be
{{GetReconfigurablePropertiesResponse}}. Similar with Request, etc. etc.
bq. DataNode's Configuration object is actually HdfsConfiguration, which sets a
few default values for certain keys. But the reconfiguration framework used
here just set the missing keys with empty / None values. Thus there are
differences.
When I type {{dfs dfsadmin -reconfig }} the
reconfiguration framework knows that I am reconfiguring a datanode. So it
should use the correct defaults for the datanode. This seems like a much
cleaner solution than hacking around this by suppressing certain valid messages
at the client.
> Reduce failure messages when running datanode reconfiguration
> -
>
> Key: HDFS-8582
> URL: https://issues.apache.org/jira/browse/HDFS-8582
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: HDFS
>Affects Versions: 2.7.0
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
>Priority: Minor
> Attachments: HDFS-8582.000.patch, HDFS-8582.001.patch,
> HDFS-8582.002.patch
>
>
> When running a DN reconfig to hotswap some drives, it spits out this output:
> {noformat}
> $ hdfs dfsadmin -reconfig datanode localhost:9023 status
> 15/06/09 14:58:10 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> Reconfiguring status for DataNode[localhost:9023]: started at Tue Jun 09
> 14:57:37 PDT 2015 and finished at Tue Jun 09 14:57:56 PDT 2015.
> FAILED: Change property
> rpc.engine.org.apache.hadoop.hdfs.protocolPB.ClientDatanodeProtocolPB
> From: "org.apache.hadoop.ipc.ProtobufRpcEngine"
> To: ""
> Error: Property
> rpc.engine.org.apache.hadoop.hdfs.protocolPB.ClientDatanodeProtocolPB is not
> reconfigurable.
> FAILED: Change property mapreduce.client.genericoptionsparser.used
> From: "true"
> To: ""
> Error: Property mapreduce.client.genericoptionsparser.used is not
> reconfigurable.
> FAILED: Change property rpc.engine.org.apache.hadoop.ipc.ProtocolMetaInfoPB
> From: "org.apache.hadoop.ipc.ProtobufRpcEngine"
> To: ""
> Error: Property rpc.engine.org.apache.hadoop.ipc.ProtocolMetaInfoPB
> is not reconfigurable.
> SUCCESS: Change property dfs.datanode.data.dir
> From: "file:///data/1/user/dfs"
> To: "file:///data/1/user/dfs,file:///data/2/user/dfs"
> FAILED: Change property dfs.datanode.startup
> From: "REGULAR"
> To: ""
> Error: Property dfs.datanode.startup is not reconfigurable.
> FAILED: Change property
> rpc.engine.org.apache.hadoop.hdfs.protocolPB.InterDatanodeProtocolPB
> From: "org.apache.hadoop.ipc.ProtobufRpcEngine"
> To: ""
> Error: Property
> rpc.engine.org.apache.hadoop.hdfs.protocolPB.InterDatanodeProtocolPB is not
> reconfigurable.
> FAILED: Change property
> rpc.engine.org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolPB
> From: "org.apache.hadoop.ipc.ProtobufRpcEngine"
> To: ""
> Error: Property
> rpc.engine.org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolPB is not
> reconfigurable.
> FAILED: Change property
> rpc.engine.org.apache.hadoop.tracing.TraceAdminProtocolPB
> From: "org.apache.hadoop.ipc.ProtobufRpcEngine"
> To: ""
> Error: Property
> rpc.engine.org.apache.hadoop.tracing.TraceAdminProtocolPB is not
> reconfigurable.
> {noformat}
> These failed messages are spurious and should not be shown.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8582) Reduce failure messages when running datanode reconfiguration
[
https://issues.apache.org/jira/browse/HDFS-8582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lei (Eddy) Xu updated HDFS-8582:
Attachment: HDFS-8582.002.patch
Updated the patch to address failure tests and checkstyle warnings.
> Reduce failure messages when running datanode reconfiguration
> -
>
> Key: HDFS-8582
> URL: https://issues.apache.org/jira/browse/HDFS-8582
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: HDFS
>Affects Versions: 2.7.0
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
>Priority: Minor
> Attachments: HDFS-8582.000.patch, HDFS-8582.001.patch,
> HDFS-8582.002.patch
>
>
> When running a DN reconfig to hotswap some drives, it spits out this output:
> {noformat}
> $ hdfs dfsadmin -reconfig datanode localhost:9023 status
> 15/06/09 14:58:10 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> Reconfiguring status for DataNode[localhost:9023]: started at Tue Jun 09
> 14:57:37 PDT 2015 and finished at Tue Jun 09 14:57:56 PDT 2015.
> FAILED: Change property
> rpc.engine.org.apache.hadoop.hdfs.protocolPB.ClientDatanodeProtocolPB
> From: "org.apache.hadoop.ipc.ProtobufRpcEngine"
> To: ""
> Error: Property
> rpc.engine.org.apache.hadoop.hdfs.protocolPB.ClientDatanodeProtocolPB is not
> reconfigurable.
> FAILED: Change property mapreduce.client.genericoptionsparser.used
> From: "true"
> To: ""
> Error: Property mapreduce.client.genericoptionsparser.used is not
> reconfigurable.
> FAILED: Change property rpc.engine.org.apache.hadoop.ipc.ProtocolMetaInfoPB
> From: "org.apache.hadoop.ipc.ProtobufRpcEngine"
> To: ""
> Error: Property rpc.engine.org.apache.hadoop.ipc.ProtocolMetaInfoPB
> is not reconfigurable.
> SUCCESS: Change property dfs.datanode.data.dir
> From: "file:///data/1/user/dfs"
> To: "file:///data/1/user/dfs,file:///data/2/user/dfs"
> FAILED: Change property dfs.datanode.startup
> From: "REGULAR"
> To: ""
> Error: Property dfs.datanode.startup is not reconfigurable.
> FAILED: Change property
> rpc.engine.org.apache.hadoop.hdfs.protocolPB.InterDatanodeProtocolPB
> From: "org.apache.hadoop.ipc.ProtobufRpcEngine"
> To: ""
> Error: Property
> rpc.engine.org.apache.hadoop.hdfs.protocolPB.InterDatanodeProtocolPB is not
> reconfigurable.
> FAILED: Change property
> rpc.engine.org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolPB
> From: "org.apache.hadoop.ipc.ProtobufRpcEngine"
> To: ""
> Error: Property
> rpc.engine.org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolPB is not
> reconfigurable.
> FAILED: Change property
> rpc.engine.org.apache.hadoop.tracing.TraceAdminProtocolPB
> From: "org.apache.hadoop.ipc.ProtobufRpcEngine"
> To: ""
> Error: Property
> rpc.engine.org.apache.hadoop.tracing.TraceAdminProtocolPB is not
> reconfigurable.
> {noformat}
> These failed messages are spurious and should not be shown.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8591) Remove support for deprecated configuration key dfs.namenode.decommission.nodes.per.interval
[ https://issues.apache.org/jira/browse/HDFS-8591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14586418#comment-14586418 ] Colin Patrick McCabe commented on HDFS-8591: +1. Thanks, [~andrew.wang]. > Remove support for deprecated configuration key > dfs.namenode.decommission.nodes.per.interval > > > Key: HDFS-8591 > URL: https://issues.apache.org/jira/browse/HDFS-8591 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.0.0 >Reporter: Andrew Wang >Assignee: Andrew Wang >Priority: Minor > Attachments: hdfs-8591.001.patch > > > dfs.namenode.decommission.nodes.per.interval is deprecated in branch-2 and > can be removed in trunk. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8456) Ozone: Introduce STORAGE_CONTAINER_SERVICE as a new NodeType.
[ https://issues.apache.org/jira/browse/HDFS-8456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-8456: Resolution: Fixed Fix Version/s: HDFS-7240 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks for the review [~jnp]. Committed to the feature branch. > Ozone: Introduce STORAGE_CONTAINER_SERVICE as a new NodeType. > - > > Key: HDFS-8456 > URL: https://issues.apache.org/jira/browse/HDFS-8456 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Fix For: HDFS-7240 > > Attachments: HDFS-8456-HDFS-7240.01.patch, > HDFS-8456-HDFS-7240.02.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8456) Ozone: Introduce STORAGE_CONTAINER_SERVICE as a new NodeType.
[ https://issues.apache.org/jira/browse/HDFS-8456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-8456: Summary: Ozone: Introduce STORAGE_CONTAINER_SERVICE as a new NodeType. (was: Introduce STORAGE_CONTAINER_SERVICE as a new NodeType.) > Ozone: Introduce STORAGE_CONTAINER_SERVICE as a new NodeType. > - > > Key: HDFS-8456 > URL: https://issues.apache.org/jira/browse/HDFS-8456 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Attachments: HDFS-8456-HDFS-7240.01.patch, > HDFS-8456-HDFS-7240.02.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8457) Ozone: Refactor FsDatasetSpi to pull up HDFS-agnostic functionality into parent interface
[ https://issues.apache.org/jira/browse/HDFS-8457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-8457: Summary: Ozone: Refactor FsDatasetSpi to pull up HDFS-agnostic functionality into parent interface (was: Refactor FsDatasetSpi to pull up HDFS-agnostic functionality into parent interface) > Ozone: Refactor FsDatasetSpi to pull up HDFS-agnostic functionality into > parent interface > - > > Key: HDFS-8457 > URL: https://issues.apache.org/jira/browse/HDFS-8457 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Attachments: HDFS-8457-HDFS-7240.01.patch, > HDFS-8457-HDFS-7240.02.patch, HDFS-8457-HDFS-7240.03.patch, > HDFS-8457-HDFS-7240.04.patch, HDFS-8457-HDFS-7240.05.patch > > > FsDatasetSpi can be split up into HDFS-specific and HDFS-agnostic parts. The > HDFS-specific parts can continue to be retained in FsDataSpi while those > relating to volume management, block pools and upgrade can be moved to a > parent interface. > There will be no change to implementations of FsDatasetSpi. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7912) Erasure Coding: track BlockInfo instead of Block in UnderReplicatedBlocks and PendingReplicationBlocks
[
https://issues.apache.org/jira/browse/HDFS-7912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14586404#comment-14586404
]
Jing Zhao commented on HDFS-7912:
-
Yeah, that will be helpful.
> Erasure Coding: track BlockInfo instead of Block in UnderReplicatedBlocks and
> PendingReplicationBlocks
> --
>
> Key: HDFS-7912
> URL: https://issues.apache.org/jira/browse/HDFS-7912
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Jing Zhao
>Assignee: Jing Zhao
> Fix For: HDFS-7285
>
> Attachments: HDFS-7912.000.patch
>
>
> Now with striped blocks and the design that uses a single BlockInfoStriped
> object to track all the corresponding blocks, we need to clearly distinguish
> the type Block and BlockInfo in BlockManager. Specifically, data structures
> like {{UnderReplicatedBlocks}} and {{PendingReplicationBlocks}} should track
> BlockInfo instead of Block in order to support striped block recovery.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8456) Introduce STORAGE_CONTAINER_SERVICE as a new NodeType.
[ https://issues.apache.org/jira/browse/HDFS-8456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14586402#comment-14586402 ] Jitendra Nath Pandey commented on HDFS-8456: +1 > Introduce STORAGE_CONTAINER_SERVICE as a new NodeType. > -- > > Key: HDFS-8456 > URL: https://issues.apache.org/jira/browse/HDFS-8456 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Attachments: HDFS-8456-HDFS-7240.01.patch, > HDFS-8456-HDFS-7240.02.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8580) Erasure coding: Persist cellSize in BlockInfoStriped and StripedBlocksFeature
[
https://issues.apache.org/jira/browse/HDFS-8580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14586398#comment-14586398
]
Jing Zhao commented on HDFS-8580:
-
The 002 patch looks good to me. One comment is that while saving an fsimage,
the {{fsn.getErasureCodingZoneForPath(n.getName())}} call is very expensive
since it has to regenerate a full path and resolve it. Thus how about just
retrieving the cell size from the INode's local striped block info?
{code}
+final int cellSize = fsn.getErasureCodingZoneForPath(n.getName())
+.getCellSize();
+builder.setCellSize(cellSize);
{code}
> Erasure coding: Persist cellSize in BlockInfoStriped and StripedBlocksFeature
> -
>
> Key: HDFS-8580
> URL: https://issues.apache.org/jira/browse/HDFS-8580
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Walter Su
>Assignee: Walter Su
> Attachments: HDFS-8580-HDFS-7285.01.patch,
> HDFS-8580-HDFS-7285.02.patch, HDFS-8580.00.patch
>
>
> Zhe Zhang, Kai Zheng and I had a offline discussion. Here is what we thought:
> Add a cellSize field in BlockInfoStriped as a workaround, and deal with
> memory usage in follow-on.(HDFS-8059)
> discussion in HDFS-8494:
> from Walter Su:
> {quote}
> I think BlockInfoStriped needs to keep cellSize.
> {quote}
> from [~vinayrpet]:
> {quote}
> I too was thinking the same when the FSImageLoader problem has came up. This
> will increase the memory usage by ~4bytes for each block though.
> {quote}
> from [~jingzhao]
> {quote}
> -Also, we should consider adding a chunk size field to StripedBlockProto and
> removing the cell size field from HdfsFileStatus. In this way we can access
> the chunk size information in the storage layer.-
> {quote}
> ==
> update:
> from [~jingzhao]
> {quote}
> For fsimage part, since HDFS-8585 just removes StripedBlockProto, I guess
> what we can do here is to either 1) add the cellSize information into
> StripedBlocksFeature in fsimage.proto, or 2) bring StripedBlockProto back and
> put block info and cell size there.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8078) HDFS client gets errors trying to to connect to IPv6 DataNode
[ https://issues.apache.org/jira/browse/HDFS-8078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14586351#comment-14586351 ] Elliott Clark commented on HDFS-8078: - I don't think that a feature branch is really needed here since each part gets things better. Lets just keep moving forward rather than one big bang integration. I'm +1 on the patch. I've seen the results on an ipv6 machine. > HDFS client gets errors trying to to connect to IPv6 DataNode > - > > Key: HDFS-8078 > URL: https://issues.apache.org/jira/browse/HDFS-8078 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Affects Versions: 2.6.0 >Reporter: Nate Edel >Assignee: Nate Edel > Labels: BB2015-05-TBR, ipv6 > Attachments: HDFS-8078.9.patch > > > 1st exception, on put: > 15/03/23 18:43:18 WARN hdfs.DFSClient: DataStreamer Exception > java.lang.IllegalArgumentException: Does not contain a valid host:port > authority: 2401:db00:1010:70ba:face:0:8:0:50010 > at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:212) > at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:164) > at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:153) > at > org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1607) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1408) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1361) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:588) > Appears to actually stem from code in DataNodeID which assumes it's safe to > append together (ipaddr + ":" + port) -- which is OK for IPv4 and not OK for > IPv6. NetUtils.createSocketAddr( ) assembles a Java URI object, which > requires the format proto://[2401:db00:1010:70ba:face:0:8:0]:50010 > Currently using InetAddress.getByName() to validate IPv6 (guava > InetAddresses.forString has been flaky) but could also use our own parsing. > (From logging this, it seems like a low-enough frequency call that the extra > object creation shouldn't be problematic, and for me the slight risk of > passing in bad input that is not actually an IPv4 or IPv6 address and thus > calling an external DNS lookup is outweighed by getting the address > normalized and avoiding rewriting parsing.) > Alternatively, sun.net.util.IPAddressUtil.isIPv6LiteralAddress() > --- > 2nd exception (on datanode) > 15/04/13 13:18:07 ERROR datanode.DataNode: > dev1903.prn1.facebook.com:50010:DataXceiver error processing unknown > operation src: /2401:db00:20:7013:face:0:7:0:54152 dst: > /2401:db00:11:d010:face:0:2f:0:50010 > java.io.EOFException > at java.io.DataInputStream.readShort(DataInputStream.java:315) > at > org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:226) > at java.lang.Thread.run(Thread.java:745) > Which also comes as client error "-get: 2401 is not an IP string literal." > This one has existing parsing logic which needs to shift to the last colon > rather than the first. Should also be a tiny bit faster by using lastIndexOf > rather than split. Could alternatively use the techniques above. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8607) TestFileCorruption doesn't work as expected
[
https://issues.apache.org/jira/browse/HDFS-8607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14586344#comment-14586344
]
Hudson commented on HDFS-8607:
--
FAILURE: Integrated in Hadoop-trunk-Commit #8017 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/8017/])
HDFS-8607. TestFileCorruption doesn't work as expected. (Contributed by Walter
Su) (arp: rev 32ffda1266c11dd98ce7f439a89fce206adf751a)
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileCorruption.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
> TestFileCorruption doesn't work as expected
> ---
>
> Key: HDFS-8607
> URL: https://issues.apache.org/jira/browse/HDFS-8607
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 2.7.0
>Reporter: Walter Su
>Assignee: Walter Su
> Fix For: 2.8.0
>
> Attachments: HDFS-8607.01.patch
>
>
> Although it passes, it's useless.
> {code}
> 77 File[] blocks = data_dir.listFiles();
> 78 assertTrue("Blocks do not exist in data-dir", (blocks != null) &&
> (blocks.length > 0));
> 79 for (int idx = 0; idx < blocks.length; idx++) {
> 80if (!blocks[idx].getName().startsWith(Block.BLOCK_FILE_PREFIX)) {
> 81 continue;
> 82}
> 83System.out.println("Deliberately removing file
> "+blocks[idx].getName());
> 84assertTrue("Cannot remove file.", blocks[idx].delete());
> 85 }
> {code}
> blocks are located at finalized/subdir0/subdir0, but line 77 only returns
> "subdir0" because {{File.listFiles()}} is not recursive. So line 83~84 will
> never be excuted.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8493) Consolidate truncate() related implementation in a single class
[
https://issues.apache.org/jira/browse/HDFS-8493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14586343#comment-14586343
]
Hadoop QA commented on HDFS-8493:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 17m 52s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 36s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 39s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 2m 12s | The applied patch generated 4
new checkstyle issues (total was 403, now 400). |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 33s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 3m 18s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 15s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 166m 11s | Tests failed in hadoop-hdfs. |
| | | 212m 34s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.hdfs.server.namenode.TestFileTruncate |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12739614/HDFS-8493-005.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 4c5da9b |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/11359/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/11359/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/11359/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/11359/console |
This message was automatically generated.
> Consolidate truncate() related implementation in a single class
> ---
>
> Key: HDFS-8493
> URL: https://issues.apache.org/jira/browse/HDFS-8493
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Haohui Mai
>Assignee: Rakesh R
> Attachments: HDFS-8493-001.patch, HDFS-8493-002.patch,
> HDFS-8493-003.patch, HDFS-8493-004.patch, HDFS-8493-005.patch
>
>
> This jira proposes to consolidate truncate() related methods into a single
> class.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8607) TestFileCorruption doesn't work as expected
[
https://issues.apache.org/jira/browse/HDFS-8607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDFS-8607:
Affects Version/s: 2.7.0
> TestFileCorruption doesn't work as expected
> ---
>
> Key: HDFS-8607
> URL: https://issues.apache.org/jira/browse/HDFS-8607
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 2.7.0
>Reporter: Walter Su
>Assignee: Walter Su
> Fix For: 2.8.0
>
> Attachments: HDFS-8607.01.patch
>
>
> Although it passes, it's useless.
> {code}
> 77 File[] blocks = data_dir.listFiles();
> 78 assertTrue("Blocks do not exist in data-dir", (blocks != null) &&
> (blocks.length > 0));
> 79 for (int idx = 0; idx < blocks.length; idx++) {
> 80if (!blocks[idx].getName().startsWith(Block.BLOCK_FILE_PREFIX)) {
> 81 continue;
> 82}
> 83System.out.println("Deliberately removing file
> "+blocks[idx].getName());
> 84assertTrue("Cannot remove file.", blocks[idx].delete());
> 85 }
> {code}
> blocks are located at finalized/subdir0/subdir0, but line 77 only returns
> "subdir0" because {{File.listFiles()}} is not recursive. So line 83~84 will
> never be excuted.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8607) TestFileCorruption doesn't work as expected
[
https://issues.apache.org/jira/browse/HDFS-8607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDFS-8607:
Component/s: test
> TestFileCorruption doesn't work as expected
> ---
>
> Key: HDFS-8607
> URL: https://issues.apache.org/jira/browse/HDFS-8607
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 2.7.0
>Reporter: Walter Su
>Assignee: Walter Su
> Fix For: 2.8.0
>
> Attachments: HDFS-8607.01.patch
>
>
> Although it passes, it's useless.
> {code}
> 77 File[] blocks = data_dir.listFiles();
> 78 assertTrue("Blocks do not exist in data-dir", (blocks != null) &&
> (blocks.length > 0));
> 79 for (int idx = 0; idx < blocks.length; idx++) {
> 80if (!blocks[idx].getName().startsWith(Block.BLOCK_FILE_PREFIX)) {
> 81 continue;
> 82}
> 83System.out.println("Deliberately removing file
> "+blocks[idx].getName());
> 84assertTrue("Cannot remove file.", blocks[idx].delete());
> 85 }
> {code}
> blocks are located at finalized/subdir0/subdir0, but line 77 only returns
> "subdir0" because {{File.listFiles()}} is not recursive. So line 83~84 will
> never be excuted.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8607) TestFileCorruption doesn't work as expected
[
https://issues.apache.org/jira/browse/HDFS-8607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDFS-8607:
Resolution: Fixed
Fix Version/s: 2.8.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Committed for 2.8.0. Thank you for reporting and fixing this [~walter.k.su].
> TestFileCorruption doesn't work as expected
> ---
>
> Key: HDFS-8607
> URL: https://issues.apache.org/jira/browse/HDFS-8607
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Walter Su
>Assignee: Walter Su
> Fix For: 2.8.0
>
> Attachments: HDFS-8607.01.patch
>
>
> Although it passes, it's useless.
> {code}
> 77 File[] blocks = data_dir.listFiles();
> 78 assertTrue("Blocks do not exist in data-dir", (blocks != null) &&
> (blocks.length > 0));
> 79 for (int idx = 0; idx < blocks.length; idx++) {
> 80if (!blocks[idx].getName().startsWith(Block.BLOCK_FILE_PREFIX)) {
> 81 continue;
> 82}
> 83System.out.println("Deliberately removing file
> "+blocks[idx].getName());
> 84assertTrue("Cannot remove file.", blocks[idx].delete());
> 85 }
> {code}
> blocks are located at finalized/subdir0/subdir0, but line 77 only returns
> "subdir0" because {{File.listFiles()}} is not recursive. So line 83~84 will
> never be excuted.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8607) TestFileCorruption doesn't work as expected
[
https://issues.apache.org/jira/browse/HDFS-8607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14586325#comment-14586325
]
Arpit Agarwal commented on HDFS-8607:
-
+1 for the patch, nice catch! Committing it shortly.
> TestFileCorruption doesn't work as expected
> ---
>
> Key: HDFS-8607
> URL: https://issues.apache.org/jira/browse/HDFS-8607
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Walter Su
>Assignee: Walter Su
> Attachments: HDFS-8607.01.patch
>
>
> Although it passes, it's useless.
> {code}
> 77 File[] blocks = data_dir.listFiles();
> 78 assertTrue("Blocks do not exist in data-dir", (blocks != null) &&
> (blocks.length > 0));
> 79 for (int idx = 0; idx < blocks.length; idx++) {
> 80if (!blocks[idx].getName().startsWith(Block.BLOCK_FILE_PREFIX)) {
> 81 continue;
> 82}
> 83System.out.println("Deliberately removing file
> "+blocks[idx].getName());
> 84assertTrue("Cannot remove file.", blocks[idx].delete());
> 85 }
> {code}
> blocks are located at finalized/subdir0/subdir0, but line 77 only returns
> "subdir0" because {{File.listFiles()}} is not recursive. So line 83~84 will
> never be excuted.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8246) Get HDFS file name based on block pool id and block id
[ https://issues.apache.org/jira/browse/HDFS-8246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14586244#comment-14586244 ] feng xu commented on HDFS-8246: --- If hdfsListDirectory() is enhanced to get HDFS file name and snapshots from block id, it will cover more use cases and benefit more people and applications. I think it’s better than fsck. > Get HDFS file name based on block pool id and block id > -- > > Key: HDFS-8246 > URL: https://issues.apache.org/jira/browse/HDFS-8246 > Project: Hadoop HDFS > Issue Type: New Feature > Components: HDFS, hdfs-client, namenode >Reporter: feng xu >Assignee: feng xu > Labels: BB2015-05-TBR > Attachments: HDFS-8246.0.patch > > > This feature provides HDFS shell command and C/Java API to retrieve HDFS file > name based on block pool id and block id. > 1. The Java API in class DistributedFileSystem > public String getFileName(String poolId, long blockId) throws IOException > 2. The C API in hdfs.c > char* hdfsGetFileName(hdfsFS fs, const char* poolId, int64_t blockId) > 3. The HDFS shell command > hdfs dfs [generic options] -fn > This feature is useful if you have HDFS block file name in local file system > and want to find out the related HDFS file name in HDFS name space > (http://stackoverflow.com/questions/10881449/how-to-find-file-from-blockname-in-hdfs-hadoop). > Each HDFS block file name in local file system contains both block pool id > and block id, for sample HDFS block file name > /hdfs/1/hadoop/hdfs/data/current/BP-97622798-10.3.11.84-1428081035160/current/finalized/subdir0/subdir0/blk_1073741825, > the block pool id is BP-97622798-10.3.11.84-1428081035160 and the block id > is 1073741825. The block pool id is uniquely related to a HDFS name > node/name space, and the block id is uniquely related to a HDFS file within > a HDFS name node/name space, so the combination of block pool id and a block > id is uniquely related a HDFS file name. > The shell command and C/Java API do not map the block pool id to name node, > so it’s user’s responsibility to talk to the correct name node in federation > environment that has multiple name nodes. The block pool id is used by name > node to check if the user is talking with the correct name node. > The implementation is straightforward. The client request to get HDFS file > name reaches the new method String getFileName(String poolId, long blockId) > in FSNamesystem in name node through RPC, and the new method does the > followings, > (1) Validate the block pool id. > (2) Create Block based on the block id. > (3) Get BlockInfoContiguous from Block. > (4) Get BlockCollection from BlockInfoContiguous. > (5) Get file name from BlockCollection. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HDFS-8547) The space reservation for RBW block is leaking
[
https://issues.apache.org/jira/browse/HDFS-8547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kihwal Lee resolved HDFS-8547.
--
Resolution: Duplicate
> The space reservation for RBW block is leaking
> --
>
> Key: HDFS-8547
> URL: https://issues.apache.org/jira/browse/HDFS-8547
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Kihwal Lee
>Priority: Critical
>
> HDFS-6898 added the feature to reserve space on datanode when creating RBW
> replicas. We have noticed that nonDfsUsed increasing on some of the
> datanodes. Heap dump of datanodes has revealed that {{reservedForRbw}} was
> huge for each volume. There weren't many rbw blocks at that time, so
> datanode must leaking it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8595) TestCommitBlockSynchronization fails in branch-2.7
[ https://issues.apache.org/jira/browse/HDFS-8595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14586189#comment-14586189 ] Hudson commented on HDFS-8595: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #2175 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2175/]) HDFS-8595. TestCommitBlockSynchronization fails in branch-2.7. (Patch applies to all branches). (Contributed by Arpit Agarwal) (arp: rev 4c5da9bd94baca4adc7200272bc2de534c39135c) * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCommitBlockSynchronization.java > TestCommitBlockSynchronization fails in branch-2.7 > -- > > Key: HDFS-8595 > URL: https://issues.apache.org/jira/browse/HDFS-8595 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 2.7.0 >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Fix For: 2.7.1 > > Attachments: HDFS-8595.01.patch > > > Mock-based TestCommitBlockSynchronization fails in branch-2.7 with NPE due to > a log statement dereferencing a null object. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8596) TestDistributedFileSystem et al tests are broken in branch-2 due to incorrect setting of "datanode" attribute
[
https://issues.apache.org/jira/browse/HDFS-8596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14586190#comment-14586190
]
Hudson commented on HDFS-8596:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk #2175 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2175/])
HDFS-8596. TestDistributedFileSystem et al tests are broken in branch-2 due to
incorrect setting of "datanode" attribute. Contributed by Yongjun Zhang.
(yzhang: rev b0dc291961410b6ac2b275cdcff4b95d75727e8d)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/DatanodeHttpServer.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
> TestDistributedFileSystem et al tests are broken in branch-2 due to incorrect
> setting of "datanode" attribute
> -
>
> Key: HDFS-8596
> URL: https://issues.apache.org/jira/browse/HDFS-8596
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: HDFS
>Affects Versions: 2.7.1
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
>Priority: Blocker
> Fix For: 2.7.1
>
> Attachments: HDFS-8596.001.patch
>
>
> After HDFS-8572, some tests failed in branch-2 as below:
> {code}
> Running org.apache.hadoop.hdfs.TestDistributedFileSystem
> Tests run: 17, Failures: 0, Errors: 3, Skipped: 0, Time elapsed: 44.302 sec
> <<< FAILURE! - in org.apache.hadoop.hdfs.TestDistributedFileSystem
> testAllWithDualPort(org.apache.hadoop.hdfs.TestDistributedFileSystem) Time
> elapsed: 7.985 sec <<< ERROR!
> java.io.IOException: Server returned HTTP response code: 500 for URL:
> http://127.0.0.1:43950/getFileChecksum/filechecksum/foo0?ugi=yzhangx&nnaddr=localhost:57316
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1625)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem$ChecksumParser.getFileChecksum(HftpFileSystem.java:528)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem$ChecksumParser.access$200(HftpFileSystem.java:504)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem.getFileChecksum(HftpFileSystem.java:545)
> at
> org.apache.hadoop.hdfs.TestDistributedFileSystem.testFileChecksum(TestDistributedFileSystem.java:592)
> at
> org.apache.hadoop.hdfs.TestDistributedFileSystem.testAllWithDualPort(TestDistributedFileSystem.java:683)
> {code}
> I found that the cause is the wrong setting of "datanode" attribute in
> ServletContext introduced by this change.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8595) TestCommitBlockSynchronization fails in branch-2.7
[ https://issues.apache.org/jira/browse/HDFS-8595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14586166#comment-14586166 ] Hudson commented on HDFS-8595: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #227 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/227/]) HDFS-8595. TestCommitBlockSynchronization fails in branch-2.7. (Patch applies to all branches). (Contributed by Arpit Agarwal) (arp: rev 4c5da9bd94baca4adc7200272bc2de534c39135c) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCommitBlockSynchronization.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt > TestCommitBlockSynchronization fails in branch-2.7 > -- > > Key: HDFS-8595 > URL: https://issues.apache.org/jira/browse/HDFS-8595 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 2.7.0 >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Fix For: 2.7.1 > > Attachments: HDFS-8595.01.patch > > > Mock-based TestCommitBlockSynchronization fails in branch-2.7 with NPE due to > a log statement dereferencing a null object. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8596) TestDistributedFileSystem et al tests are broken in branch-2 due to incorrect setting of "datanode" attribute
[
https://issues.apache.org/jira/browse/HDFS-8596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14586167#comment-14586167
]
Hudson commented on HDFS-8596:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #227 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/227/])
HDFS-8596. TestDistributedFileSystem et al tests are broken in branch-2 due to
incorrect setting of "datanode" attribute. Contributed by Yongjun Zhang.
(yzhang: rev b0dc291961410b6ac2b275cdcff4b95d75727e8d)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/DatanodeHttpServer.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
> TestDistributedFileSystem et al tests are broken in branch-2 due to incorrect
> setting of "datanode" attribute
> -
>
> Key: HDFS-8596
> URL: https://issues.apache.org/jira/browse/HDFS-8596
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: HDFS
>Affects Versions: 2.7.1
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
>Priority: Blocker
> Fix For: 2.7.1
>
> Attachments: HDFS-8596.001.patch
>
>
> After HDFS-8572, some tests failed in branch-2 as below:
> {code}
> Running org.apache.hadoop.hdfs.TestDistributedFileSystem
> Tests run: 17, Failures: 0, Errors: 3, Skipped: 0, Time elapsed: 44.302 sec
> <<< FAILURE! - in org.apache.hadoop.hdfs.TestDistributedFileSystem
> testAllWithDualPort(org.apache.hadoop.hdfs.TestDistributedFileSystem) Time
> elapsed: 7.985 sec <<< ERROR!
> java.io.IOException: Server returned HTTP response code: 500 for URL:
> http://127.0.0.1:43950/getFileChecksum/filechecksum/foo0?ugi=yzhangx&nnaddr=localhost:57316
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1625)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem$ChecksumParser.getFileChecksum(HftpFileSystem.java:528)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem$ChecksumParser.access$200(HftpFileSystem.java:504)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem.getFileChecksum(HftpFileSystem.java:545)
> at
> org.apache.hadoop.hdfs.TestDistributedFileSystem.testFileChecksum(TestDistributedFileSystem.java:592)
> at
> org.apache.hadoop.hdfs.TestDistributedFileSystem.testAllWithDualPort(TestDistributedFileSystem.java:683)
> {code}
> I found that the cause is the wrong setting of "datanode" attribute in
> ServletContext introduced by this change.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8586) Dead Datanode is allocated for write when client is from deadnode
[
https://issues.apache.org/jira/browse/HDFS-8586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14586118#comment-14586118
]
Brahma Reddy Battula commented on HDFS-8586:
Workaround can be done by enabling " *dfs.name.avoid.write.stale.datanode* "
where node is considered as stale( if there is no heartbeat in 30 sec's
bydefault)..such that it's not allocated for the write...Anyone have some other
thoughts..?
> Dead Datanode is allocated for write when client is from deadnode
> --
>
> Key: HDFS-8586
> URL: https://issues.apache.org/jira/browse/HDFS-8586
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
>Priority: Critical
>
> *{color:blue}DataNode marked as Dead{color}*
> 2015-06-11 19:39:00,862 | INFO |
> org.apache.hadoop.hdfs.server.blockmanagement.HeartbeatManager$Monitor@28ec166e
> | BLOCK* *removeDeadDatanode: lost heartbeat from XX.XX.39.33:25009* |
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.removeDeadDatanode(DatanodeManager.java:584)
> 2015-06-11 19:39:00,863 | INFO |
> org.apache.hadoop.hdfs.server.blockmanagement.HeartbeatManager$Monitor@28ec166e
> | Removing a node: /default/rack3/XX.XX.39.33:25009 |
> org.apache.hadoop.net.NetworkTopology.remove(NetworkTopology.java:488)
> *{color:blue}Deadnode got Allocated{color}*
> 2015-06-11 19:39:45,148 | WARN | IPC Server handler 26 on 25000 | The
> cluster does not contain node: /default/rack3/XX.XX.39.33:25009 |
> org.apache.hadoop.net.NetworkTopology.getDistance(NetworkTopology.java:616)
> 2015-06-11 19:39:45,149 | WARN | IPC Server handler 26 on 25000 | The
> cluster does not contain node: /default/rack3/XX.XX.39.33:25009 |
> org.apache.hadoop.net.NetworkTopology.getDistance(NetworkTopology.java:616)
> 2015-06-11 19:39:45,149 | WARN | IPC Server handler 26 on 25000 | The
> cluster does not contain node: /default/rack3/XX.XX.39.33:25009 |
> org.apache.hadoop.net.NetworkTopology.getDistance(NetworkTopology.java:616)
> 2015-06-11 19:39:45,149 | WARN | IPC Server handler 26 on 25000 | The
> cluster does not contain node: /default/rack3/XX.XX.39.33:25009 |
> org.apache.hadoop.net.NetworkTopology.getDistance(NetworkTopology.java:616)
> 2015-06-11 19:39:45,149 | INFO | IPC Server handler 26 on 25000 | BLOCK*
> *allocate blk_1073754030_13252* {UCState=UNDER_CONSTRUCTION,
> truncateBlock=null, primaryNodeIndex=-1,
> replicas=[ReplicaUC[[DISK]DS-e8d29773-dfc2-4224-b1d6-9b0588bca55e:NORMAL:{color:red}XX.XX.39.33:25009{color}|RBW],
>
> ReplicaUC[[DISK]DS-f7d2ab3c-88f7-470c-9097-84387c0bec83:NORMAL:XX.XX.38.32:25009|RBW],
> ReplicaUC[[DISK]DS-8c2a464a-ac81-4651-890a-dbfd07ddd95f:NORMAL:
> *XX.XX.38.33:25009|RBW]]* } for /t1._COPYING_ |
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.saveAllocatedBlock(FSNamesystem.java:3657)
> 2015-06-11 19:39:45,191 | INFO | IPC Server handler 35 on 25000 | BLOCK*
> allocate blk_1073754031_13253{UCState=UNDER_CONSTRUCTION, truncateBlock=null,
> primaryNodeIndex=-1,
> replicas=[ReplicaUC[[DISK]DS-ed8ad579-50c0-4e3e-8780-9776531763b6:NORMAL:XX.XX.39.31:25009|RBW],
>
> ReplicaUC[[DISK]DS-19ddd6da-4a3e-481a-8445-dde5c90aaff3:NORMAL:XX.XX.37.32:25009|RBW],
> ReplicaUC[[DISK]DS-4ce4ce39-4973-42ce-8c7d-cb41f899db85:
> {{NORMAL:XX.XX.37.33:25009}} |RBW]]} for /t1._COPYING_ |
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.saveAllocatedBlock(FSNamesystem.java:3657)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8595) TestCommitBlockSynchronization fails in branch-2.7
[ https://issues.apache.org/jira/browse/HDFS-8595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14586111#comment-14586111 ] Hudson commented on HDFS-8595: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk #2157 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2157/]) HDFS-8595. TestCommitBlockSynchronization fails in branch-2.7. (Patch applies to all branches). (Contributed by Arpit Agarwal) (arp: rev 4c5da9bd94baca4adc7200272bc2de534c39135c) * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCommitBlockSynchronization.java > TestCommitBlockSynchronization fails in branch-2.7 > -- > > Key: HDFS-8595 > URL: https://issues.apache.org/jira/browse/HDFS-8595 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 2.7.0 >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Fix For: 2.7.1 > > Attachments: HDFS-8595.01.patch > > > Mock-based TestCommitBlockSynchronization fails in branch-2.7 with NPE due to > a log statement dereferencing a null object. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8596) TestDistributedFileSystem et al tests are broken in branch-2 due to incorrect setting of "datanode" attribute
[
https://issues.apache.org/jira/browse/HDFS-8596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14586112#comment-14586112
]
Hudson commented on HDFS-8596:
--
SUCCESS: Integrated in Hadoop-Hdfs-trunk #2157 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/2157/])
HDFS-8596. TestDistributedFileSystem et al tests are broken in branch-2 due to
incorrect setting of "datanode" attribute. Contributed by Yongjun Zhang.
(yzhang: rev b0dc291961410b6ac2b275cdcff4b95d75727e8d)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/DatanodeHttpServer.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
> TestDistributedFileSystem et al tests are broken in branch-2 due to incorrect
> setting of "datanode" attribute
> -
>
> Key: HDFS-8596
> URL: https://issues.apache.org/jira/browse/HDFS-8596
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: HDFS
>Affects Versions: 2.7.1
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
>Priority: Blocker
> Fix For: 2.7.1
>
> Attachments: HDFS-8596.001.patch
>
>
> After HDFS-8572, some tests failed in branch-2 as below:
> {code}
> Running org.apache.hadoop.hdfs.TestDistributedFileSystem
> Tests run: 17, Failures: 0, Errors: 3, Skipped: 0, Time elapsed: 44.302 sec
> <<< FAILURE! - in org.apache.hadoop.hdfs.TestDistributedFileSystem
> testAllWithDualPort(org.apache.hadoop.hdfs.TestDistributedFileSystem) Time
> elapsed: 7.985 sec <<< ERROR!
> java.io.IOException: Server returned HTTP response code: 500 for URL:
> http://127.0.0.1:43950/getFileChecksum/filechecksum/foo0?ugi=yzhangx&nnaddr=localhost:57316
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1625)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem$ChecksumParser.getFileChecksum(HftpFileSystem.java:528)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem$ChecksumParser.access$200(HftpFileSystem.java:504)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem.getFileChecksum(HftpFileSystem.java:545)
> at
> org.apache.hadoop.hdfs.TestDistributedFileSystem.testFileChecksum(TestDistributedFileSystem.java:592)
> at
> org.apache.hadoop.hdfs.TestDistributedFileSystem.testAllWithDualPort(TestDistributedFileSystem.java:683)
> {code}
> I found that the cause is the wrong setting of "datanode" attribute in
> ServletContext introduced by this change.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8596) TestDistributedFileSystem et al tests are broken in branch-2 due to incorrect setting of "datanode" attribute
[
https://issues.apache.org/jira/browse/HDFS-8596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14586107#comment-14586107
]
Hudson commented on HDFS-8596:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #218 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/218/])
HDFS-8596. TestDistributedFileSystem et al tests are broken in branch-2 due to
incorrect setting of "datanode" attribute. Contributed by Yongjun Zhang.
(yzhang: rev b0dc291961410b6ac2b275cdcff4b95d75727e8d)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/DatanodeHttpServer.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
> TestDistributedFileSystem et al tests are broken in branch-2 due to incorrect
> setting of "datanode" attribute
> -
>
> Key: HDFS-8596
> URL: https://issues.apache.org/jira/browse/HDFS-8596
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: HDFS
>Affects Versions: 2.7.1
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
>Priority: Blocker
> Fix For: 2.7.1
>
> Attachments: HDFS-8596.001.patch
>
>
> After HDFS-8572, some tests failed in branch-2 as below:
> {code}
> Running org.apache.hadoop.hdfs.TestDistributedFileSystem
> Tests run: 17, Failures: 0, Errors: 3, Skipped: 0, Time elapsed: 44.302 sec
> <<< FAILURE! - in org.apache.hadoop.hdfs.TestDistributedFileSystem
> testAllWithDualPort(org.apache.hadoop.hdfs.TestDistributedFileSystem) Time
> elapsed: 7.985 sec <<< ERROR!
> java.io.IOException: Server returned HTTP response code: 500 for URL:
> http://127.0.0.1:43950/getFileChecksum/filechecksum/foo0?ugi=yzhangx&nnaddr=localhost:57316
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1625)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem$ChecksumParser.getFileChecksum(HftpFileSystem.java:528)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem$ChecksumParser.access$200(HftpFileSystem.java:504)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem.getFileChecksum(HftpFileSystem.java:545)
> at
> org.apache.hadoop.hdfs.TestDistributedFileSystem.testFileChecksum(TestDistributedFileSystem.java:592)
> at
> org.apache.hadoop.hdfs.TestDistributedFileSystem.testAllWithDualPort(TestDistributedFileSystem.java:683)
> {code}
> I found that the cause is the wrong setting of "datanode" attribute in
> ServletContext introduced by this change.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8595) TestCommitBlockSynchronization fails in branch-2.7
[ https://issues.apache.org/jira/browse/HDFS-8595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14586106#comment-14586106 ] Hudson commented on HDFS-8595: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #218 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/218/]) HDFS-8595. TestCommitBlockSynchronization fails in branch-2.7. (Patch applies to all branches). (Contributed by Arpit Agarwal) (arp: rev 4c5da9bd94baca4adc7200272bc2de534c39135c) * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCommitBlockSynchronization.java > TestCommitBlockSynchronization fails in branch-2.7 > -- > > Key: HDFS-8595 > URL: https://issues.apache.org/jira/browse/HDFS-8595 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 2.7.0 >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Fix For: 2.7.1 > > Attachments: HDFS-8595.01.patch > > > Mock-based TestCommitBlockSynchronization fails in branch-2.7 with NPE due to > a log statement dereferencing a null object. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8493) Consolidate truncate() related implementation in a single class
[
https://issues.apache.org/jira/browse/HDFS-8493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14586053#comment-14586053
]
Rakesh R commented on HDFS-8493:
Thank you [~wheat9] for the review comments. Attached patch addressing the
comments.
bq. Most of the functions in FSDirTruncateOp should be private instead of
package local.
Since FsDirTruncateOp#prepareFileForTruncate is used in tests, have made
{{@VisibleForTesting}} to it.
> Consolidate truncate() related implementation in a single class
> ---
>
> Key: HDFS-8493
> URL: https://issues.apache.org/jira/browse/HDFS-8493
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Haohui Mai
>Assignee: Rakesh R
> Attachments: HDFS-8493-001.patch, HDFS-8493-002.patch,
> HDFS-8493-003.patch, HDFS-8493-004.patch, HDFS-8493-005.patch
>
>
> This jira proposes to consolidate truncate() related methods into a single
> class.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
