[jira] [Commented] (HDFS-8889) Erasure Coding: cover more test situations of datanode failure during client writing
[ https://issues.apache.org/jira/browse/HDFS-8889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14693047#comment-14693047 ] Li Bo commented on HDFS-8889: - Currently 9 streamers are working together for the client writing. A small number of failed datanodes (<= 3) for a block group should not influence the writing. There’re a lot of datanode failure cases and we should cover as many as possible in unit test. Suppose streamer 4 fails, the following situations for the next block group should be considered: 1) all streamers succeed 2) Streamer 4 still fails 3) only streamer 1 fails 4) only streamer 8 fails (test parity streamer) 5) streamer 4 and 6 fail 6) streamer 4 and 1,6 fail 7) streamer 4 and 1,2,6 fail 8) streamer 2, 6 fail Suppose streamer 2 and 4 fail, the following situations for the next block group should be considered: 1) only streamer 2 and 4 fail 2) streamer 2, 4, 8 fail 3) only streamer 2 fails 4) streamer 3 , 8 fail For a single streamer, we should consider the following situations of the time of datanode failure: 1) before writing the first byte 2) before finishing writing the first cell 3) right after finishing writing the first cell 4) before writing the last byte of the block Other situations: 1) more than 3 streamers fail at the first block group 2) more than 3 streamers fail at the last block group > Erasure Coding: cover more test situations of datanode failure during client > writing > > > Key: HDFS-8889 > URL: https://issues.apache.org/jira/browse/HDFS-8889 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Li Bo >Assignee: Li Bo > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8889) Erasure Coding: cover more test situations of datanode failure during client writing
[ https://issues.apache.org/jira/browse/HDFS-8889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Bo updated HDFS-8889: Description: Currently 9 streamers are working together for the client writing. A small number of failed datanodes (<= 3) for a block group should not influence the writing. There’re a lot of datanode failure cases and we should cover as many as possible in unit test. Suppose streamer 4 fails, the following situations for the next block group should be considered: 1) all streamers succeed 2) Streamer 4 still fails 3) only streamer 1 fails 4) only streamer 8 fails (test parity streamer) 5) streamer 4 and 6 fail 6) streamer 4 and 1,6 fail 7) streamer 4 and 1,2,6 fail 8) streamer 2, 6 fail Suppose streamer 2 and 4 fail, the following situations for the next block group should be considered: 1) only streamer 2 and 4 fail 2) streamer 2, 4, 8 fail 3) only streamer 2 fails 4) streamer 3 , 8 fail For a single streamer, we should consider the following situations of the time of datanode failure: 1) before writing the first byte 2) before finishing writing the first cell 3) right after finishing writing the first cell 4) before writing the last byte of the block Other situations: 1) more than 3 streamers fail at the first block group 2) more than 3 streamers fail at the last block group > Erasure Coding: cover more test situations of datanode failure during client > writing > > > Key: HDFS-8889 > URL: https://issues.apache.org/jira/browse/HDFS-8889 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Li Bo >Assignee: Li Bo > > Currently 9 streamers are working together for the client writing. A small > number of failed datanodes (<= 3) for a block group should not influence the > writing. There’re a lot of datanode failure cases and we should cover as many > as possible in unit test. > Suppose streamer 4 fails, the following situations for the next block group > should be considered: > 1)all streamers succeed > 2)Streamer 4 still fails > 3)only streamer 1 fails > 4)only streamer 8 fails (test parity streamer) > 5)streamer 4 and 6 fail > 6)streamer 4 and 1,6 fail > 7)streamer 4 and 1,2,6 fail > 8)streamer 2, 6 fail > Suppose streamer 2 and 4 fail, the following situations for the next block > group should be considered: > 1)only streamer 2 and 4 fail > 2)streamer 2, 4, 8 fail > 3)only streamer 2 fails > 4)streamer 3 , 8 fail > For a single streamer, we should consider the following situations of the > time of datanode failure: > 1)before writing the first byte > 2)before finishing writing the first cell > 3)right after finishing writing the first cell > 4)before writing the last byte of the block > Other situations: > 1)more than 3 streamers fail at the first block group > 2)more than 3 streamers fail at the last block group > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8887) Expose storage type and storage ID in BlockLocation
[ https://issues.apache.org/jira/browse/HDFS-8887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14693052#comment-14693052 ] Hudson commented on HDFS-8887: -- FAILURE: Integrated in Hadoop-trunk-Commit #8287 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/8287/]) HDFS-8887. Expose storage type and storage ID in BlockLocation. (wang: rev 1ea1a8334ea01814121490a5bfd2a0205c66d6e4) * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestBlockLocation.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/fs/BlockStorageLocation.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java * hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSUtilClient.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/BlockLocation.java > Expose storage type and storage ID in BlockLocation > --- > > Key: HDFS-8887 > URL: https://issues.apache.org/jira/browse/HDFS-8887 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 2.7.1 >Reporter: Andrew Wang >Assignee: Andrew Wang > Fix For: 2.8.0 > > Attachments: HDFS-8887.001.patch, HDFS-8887.002.patch > > > Some applications schedule based on info like storage type or storage ID, > it'd be useful to expose this information in BlockLocation. It's already > included in LocatedBlock and sent over the wire. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8854) Erasure coding: add ECPolicy to replace schema+cellSize in hadoop-hdfs
[
https://issues.apache.org/jira/browse/HDFS-8854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14693067#comment-14693067
]
Hadoop QA commented on HDFS-8854:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12750039/HDFS-8854-HDFS-7285-merge.03.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | HDFS-7285 / fbf7e81 |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/11978/console |
This message was automatically generated.
> Erasure coding: add ECPolicy to replace schema+cellSize in hadoop-hdfs
> --
>
> Key: HDFS-8854
> URL: https://issues.apache.org/jira/browse/HDFS-8854
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: HDFS-7285
>Reporter: Walter Su
>Assignee: Walter Su
> Attachments: HDFS-8854-Consolidated-20150806.02.txt,
> HDFS-8854-HDFS-7285-merge.03.patch, HDFS-8854-HDFS-7285-merge.03.txt,
> HDFS-8854-HDFS-7285.00.patch, HDFS-8854-HDFS-7285.01.patch,
> HDFS-8854-HDFS-7285.02.patch, HDFS-8854-HDFS-7285.03.patch, HDFS-8854.00.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8287) DFSStripedOutputStream.writeChunk should not wait for writing parity
[ https://issues.apache.org/jira/browse/HDFS-8287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Sasaki updated HDFS-8287: - Attachment: HDFS-8287-HDFS-7285.02.patch > DFSStripedOutputStream.writeChunk should not wait for writing parity > - > > Key: HDFS-8287 > URL: https://issues.apache.org/jira/browse/HDFS-8287 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Tsz Wo Nicholas Sze >Assignee: Kai Sasaki > Attachments: HDFS-8287-HDFS-7285.00.patch, > HDFS-8287-HDFS-7285.01.patch, HDFS-8287-HDFS-7285.02.patch > > > When a stripping cell is full, writeChunk computes and generates parity > packets. It sequentially calls waitAndQueuePacket so that user client cannot > continue to write data until it finishes. > We should allow user client to continue writing instead but not blocking it > when writing parity. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8854) Erasure coding: add ECPolicy to replace schema+cellSize in hadoop-hdfs
[
https://issues.apache.org/jira/browse/HDFS-8854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14693093#comment-14693093
]
Rakesh R commented on HDFS-8854:
While applying patch jenkins is not picking the {{HDFS-7285-merge}} branch. It
is still going to {{HDFS-7285}} branch and failing.
{code}
Switched to branch 'HDFS-7285'
Your branch is behind 'origin/HDFS-7285' by 1 commit, and can be fast-forwarded.
(use "git pull" to update your local branch)
First, rewinding head to replay your work on top of it...
Fast-forwarded HDFS-7285 to fbf7e81ca007e009b492e3b99060bbfb74394f46.
Testing HDFS-8854 patch on HDFS-7285.
{code}
> Erasure coding: add ECPolicy to replace schema+cellSize in hadoop-hdfs
> --
>
> Key: HDFS-8854
> URL: https://issues.apache.org/jira/browse/HDFS-8854
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: HDFS-7285
>Reporter: Walter Su
>Assignee: Walter Su
> Attachments: HDFS-8854-Consolidated-20150806.02.txt,
> HDFS-8854-HDFS-7285-merge.03.patch, HDFS-8854-HDFS-7285-merge.03.txt,
> HDFS-8854-HDFS-7285.00.patch, HDFS-8854-HDFS-7285.01.patch,
> HDFS-8854-HDFS-7285.02.patch, HDFS-8854-HDFS-7285.03.patch, HDFS-8854.00.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8287) DFSStripedOutputStream.writeChunk should not wait for writing parity
[
https://issues.apache.org/jira/browse/HDFS-8287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14693096#comment-14693096
]
Kai Sasaki commented on HDFS-8287:
--
[~rakeshr] Thank you so much for great advice!
{quote}
Is that your intention is to block the cellBufferses(currentIndex) till
writeParity() function is completed, considering that encoder will take more
time than next ParityGenerator#enqueue arrives. If yes, then I think we should
make ParityGenerator#enqueue() function also synchronized on bufferQueue. If
not, then will wrap only bufferQueue.wait();, right?
{quote}
Yes. It is possible to be enqueued while generating parity in
{{ParityGenerator}}. So I also think it is necessary to synchronize
{{bufferQueue}} when enqueueing. I updated.
> DFSStripedOutputStream.writeChunk should not wait for writing parity
> -
>
> Key: HDFS-8287
> URL: https://issues.apache.org/jira/browse/HDFS-8287
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Kai Sasaki
> Attachments: HDFS-8287-HDFS-7285.00.patch,
> HDFS-8287-HDFS-7285.01.patch, HDFS-8287-HDFS-7285.02.patch
>
>
> When a stripping cell is full, writeChunk computes and generates parity
> packets. It sequentially calls waitAndQueuePacket so that user client cannot
> continue to write data until it finishes.
> We should allow user client to continue writing instead but not blocking it
> when writing parity.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8388) Time and Date format need to be in sync in Namenode UI page
[
https://issues.apache.org/jira/browse/HDFS-8388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14693111#comment-14693111
]
Akira AJISAKA commented on HDFS-8388:
-
I'm thinking moment-timezone.js can parse the format by using {{user.timezone}}
property via JMX, however, it needs a lot of work. I prefer adding a new metric
rather than parsing the date. Please correct me if I am wrong.
By the way, v3 patch needs documentation for the new metric.
> Time and Date format need to be in sync in Namenode UI page
> ---
>
> Key: HDFS-8388
> URL: https://issues.apache.org/jira/browse/HDFS-8388
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Archana T
>Assignee: Surendra Singh Lilhore
>Priority: Minor
> Attachments: HDFS-8388-002.patch, HDFS-8388-003.patch,
> HDFS-8388.patch, HDFS-8388_1.patch, ScreenShot-InvalidDate.png
>
>
> In NameNode UI Page, Date and Time FORMAT displayed on the page are not in
> sync currently.
> Started:Wed May 13 12:28:02 IST 2015
> Compiled:23 Apr 2015 12:22:59
> Block Deletion Start Time 13 May 2015 12:28:02
> We can keep a common format in all the above places.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8863) The remaining space check in BlockPlacementPolicyDefault is flawed
[
https://issues.apache.org/jira/browse/HDFS-8863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA updated HDFS-8863:
Summary: The remaining space check in BlockPlacementPolicyDefault is flawed
(was: The remiaing space check in BlockPlacementPolicyDefault is flawed)
> The remaining space check in BlockPlacementPolicyDefault is flawed
> --
>
> Key: HDFS-8863
> URL: https://issues.apache.org/jira/browse/HDFS-8863
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: Kihwal Lee
>Assignee: Kihwal Lee
>Priority: Critical
> Labels: 2.6.1-candidate
> Attachments: HDFS-8863.patch, HDFS-8863.v2.patch
>
>
> The block placement policy calls
> {{DatanodeDescriptor#getRemaining(StorageType to check whether the block
> is going to fit. Since the method is adding up all remaining spaces, namenode
> can allocate a new block on a full node. This causes pipeline construction
> failure and {{abandonBlock}}. If the cluster is nearly full, the client might
> hit this multiple times and the write can fail permanently.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8852) HDFS architecture documentation of version 2.x is outdated about append write support
[ https://issues.apache.org/jira/browse/HDFS-8852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14693200#comment-14693200 ] Akira AJISAKA commented on HDFS-8852: - Thanks [~ajithshetty] for updating the patch. bq. except for appends truncate, introduced in Hadoop 2.7.0, can change the file. Would you change "appends" to "appends and truncates"? I'm +1 if that is addressed. > HDFS architecture documentation of version 2.x is outdated about append write > support > - > > Key: HDFS-8852 > URL: https://issues.apache.org/jira/browse/HDFS-8852 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation >Reporter: Hong Dai Thanh >Assignee: Ajith S > Labels: newbie > Attachments: HDFS-8852.patch > > > In the [latest version of the > documentation|http://hadoop.apache.org/docs/current2/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html#Simple_Coherency_Model], > and also documentation for all releases with version 2, it’s mentioned that > “A file once created, written, and closed need not be changed. “ and “There > is a plan to support appending-writes to files in the future.” > > However, as far as I know, HDFS has supported append write since 0.21, based > on [HDFS-265|https://issues.apache.org/jira/browse/HDFS-265] and [the old > version of the documentation in > 2012|https://web.archive.org/web/20121221171824/http://hadoop.apache.org/docs/hdfs/current/hdfs_design.html#Appending-Writes+and+File+Syncs] > Various posts on the Internet also suggests that append write has been > available in HDFS, and will always be available in Hadoop version 2 branch. > > Can we update the documentation to reflect the current status? > (Please also review whether the documentation should also be updated for > version 0.21 and above, and the version 1.x branch) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8622) Implement GETCONTENTSUMMARY operation for WebImageViewer
[
https://issues.apache.org/jira/browse/HDFS-8622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14693223#comment-14693223
]
Akira AJISAKA commented on HDFS-8622:
-
Thanks [~jagadesh.kiran] for updating the patch.
{code}
hdfs.createSymlink(new Path("parentDir/file4"), link1, true);
{code}
This path is "parentDir/file4", not equal to the path of file4
("/parentDir/file4"). Would you add "/" before "parentDir/file4" or re-use
{{file1OnParentDir}} instead of creating the same instance?
> Implement GETCONTENTSUMMARY operation for WebImageViewer
>
>
> Key: HDFS-8622
> URL: https://issues.apache.org/jira/browse/HDFS-8622
> Project: Hadoop HDFS
> Issue Type: New Feature
>Reporter: Jagadesh Kiran N
>Assignee: Jagadesh Kiran N
> Attachments: HDFS-8622-00.patch, HDFS-8622-01.patch,
> HDFS-8622-02.patch, HDFS-8622-03.patch, HDFS-8622-04.patch,
> HDFS-8622-05.patch, HDFS-8622-06.patch, HDFS-8622-07.patch,
> HDFS-8622-08.patch, HDFS-8622-09.patch
>
>
> it would be better for administrators if {code} GETCONTENTSUMMARY {code} are
> supported.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8808) dfs.image.transfer.bandwidthPerSec should not apply to -bootstrapStandby
[
https://issues.apache.org/jira/browse/HDFS-8808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14693226#comment-14693226
]
Hadoop QA commented on HDFS-8808:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 15m 29s | Findbugs (version ) appears to
be broken on trunk. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 2 new or modified test files. |
| {color:green}+1{color} | javac | 7m 52s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 50s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 47s | There were no new checkstyle
issues. |
| {color:red}-1{color} | whitespace | 0m 0s | The patch has 1 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 26s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 36s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 12s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 75m 15s | Tests failed in hadoop-hdfs. |
| | | 117m 27s | |
\\
\\
|| Reason || Tests ||
| Timed out tests | org.apache.hadoop.hdfs.TestReplaceDatanodeOnFailure |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12750029/HDFS-8808-02.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 1ea1a83 |
| whitespace |
https://builds.apache.org/job/PreCommit-HDFS-Build/11979/artifact/patchprocess/whitespace.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/11979/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/11979/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/11979/console |
This message was automatically generated.
> dfs.image.transfer.bandwidthPerSec should not apply to -bootstrapStandby
>
>
> Key: HDFS-8808
> URL: https://issues.apache.org/jira/browse/HDFS-8808
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.1
>Reporter: Gautam Gopalakrishnan
>Assignee: Zhe Zhang
> Attachments: HDFS-8808-00.patch, HDFS-8808-01.patch,
> HDFS-8808-02.patch
>
>
> The parameter {{dfs.image.transfer.bandwidthPerSec}} can be used to limit the
> speed with which the fsimage is copied between the namenodes during regular
> use. However, as a side effect, this also limits transfers when the
> {{-bootstrapStandby}} option is used. This option is often used during
> upgrades and could potentially slow down the entire workflow. The request
> here is to ensure {{-bootstrapStandby}} is unaffected by this bandwidth
> setting
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8078) HDFS client gets errors trying to to connect to IPv6 DataNode
[
https://issues.apache.org/jira/browse/HDFS-8078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14693339#comment-14693339
]
Hadoop QA commented on HDFS-8078:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 17m 20s | Findbugs (version ) appears to
be broken on trunk. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 2 new or modified test files. |
| {color:red}-1{color} | javac | 7m 42s | The applied patch generated 1
additional warning messages. |
| {color:green}+1{color} | javadoc | 9m 32s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 43s | The applied patch generated 1
new checkstyle issues (total was 39, now 38). |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 35s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 6m 10s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:red}-1{color} | common tests | 22m 12s | Tests failed in
hadoop-common. |
| {color:red}-1{color} | hdfs tests | 172m 41s | Tests failed in hadoop-hdfs. |
| {color:green}+1{color} | hdfs tests | 0m 26s | Tests passed in
hadoop-hdfs-client. |
| | | 240m 52s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.net.TestNetUtils |
| | hadoop.ha.TestZKFailoverController |
| Timed out tests | org.apache.hadoop.cli.TestHDFSCLI |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12750030/HDFS-8078.14.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 1ea1a83 |
| javac |
https://builds.apache.org/job/PreCommit-HDFS-Build/11977/artifact/patchprocess/diffJavacWarnings.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/11977/artifact/patchprocess/diffcheckstylehadoop-common.txt
|
| hadoop-common test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/11977/artifact/patchprocess/testrun_hadoop-common.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/11977/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| hadoop-hdfs-client test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/11977/artifact/patchprocess/testrun_hadoop-hdfs-client.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/11977/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/11977/console |
This message was automatically generated.
> HDFS client gets errors trying to to connect to IPv6 DataNode
> -
>
> Key: HDFS-8078
> URL: https://issues.apache.org/jira/browse/HDFS-8078
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Affects Versions: 2.6.0
>Reporter: Nate Edel
>Assignee: Nate Edel
> Labels: BB2015-05-TBR, ipv6
> Attachments: HDFS-8078.10.patch, HDFS-8078.11.patch,
> HDFS-8078.12.patch, HDFS-8078.13.patch, HDFS-8078.14.patch, HDFS-8078.9.patch
>
>
> 1st exception, on put:
> 15/03/23 18:43:18 WARN hdfs.DFSClient: DataStreamer Exception
> java.lang.IllegalArgumentException: Does not contain a valid host:port
> authority: 2401:db00:1010:70ba:face:0:8:0:50010
> at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:212)
> at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:164)
> at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:153)
> at
> org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1607)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1408)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1361)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:588)
> Appears to actually stem from code in DataNodeID which assumes it's safe to
> append together (ipaddr + ":" + port) -- which is OK for IPv4 and not OK for
> IPv6. NetUtils.createSocketAddr( ) assembles a Java URI object, which
> requires the format prot
[jira] [Commented] (HDFS-8859) Improve DataNode (ReplicaMap) memory footprint to save about 45%
[
https://issues.apache.org/jira/browse/HDFS-8859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14693340#comment-14693340
]
Hadoop QA commented on HDFS-8859:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 17m 2s | Findbugs (version ) appears to
be broken on trunk. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 3 new or modified test files. |
| {color:green}+1{color} | javac | 7m 44s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 38s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 29s | The applied patch generated 6
new checkstyle issues (total was 12, now 14). |
| {color:red}-1{color} | whitespace | 0m 1s | The patch has 3 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 29s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 4m 25s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:red}-1{color} | common tests | 22m 16s | Tests failed in
hadoop-common. |
| {color:red}-1{color} | hdfs tests | 220m 50s | Tests failed in hadoop-hdfs. |
| | | 286m 7s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.ha.TestZKFailoverController |
| | hadoop.net.TestNetUtils |
| | hadoop.hdfs.server.namenode.ha.TestDNFencing |
| | hadoop.hdfs.server.namenode.snapshot.TestSnapshot |
| Timed out tests |
org.apache.hadoop.hdfs.server.namenode.TestParallelImageWrite |
| | org.apache.hadoop.hdfs.server.namenode.TestFsck |
| | org.apache.hadoop.hdfs.TestRestartDFS |
| | org.apache.hadoop.cli.TestHDFSCLI |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12750016/HDFS-8859.003.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 1ea1a83 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/11976/artifact/patchprocess/diffcheckstylehadoop-common.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-HDFS-Build/11976/artifact/patchprocess/whitespace.txt
|
| hadoop-common test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/11976/artifact/patchprocess/testrun_hadoop-common.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/11976/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/11976/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/11976/console |
This message was automatically generated.
> Improve DataNode (ReplicaMap) memory footprint to save about 45%
>
>
> Key: HDFS-8859
> URL: https://issues.apache.org/jira/browse/HDFS-8859
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Reporter: Yi Liu
>Assignee: Yi Liu
>Priority: Critical
> Attachments: HDFS-8859.001.patch, HDFS-8859.002.patch,
> HDFS-8859.003.patch
>
>
> By using following approach we can save about *45%* memory footprint for each
> block replica in DataNode memory (This JIRA only talks about *ReplicaMap* in
> DataNode), the details are:
> In ReplicaMap,
> {code}
> private final Map> map =
> new HashMap>();
> {code}
> Currently we use a HashMap {{Map}} to store the replicas
> in memory. The key is block id of the block replica which is already
> included in {{ReplicaInfo}}, so this memory can be saved. Also HashMap Entry
> has a object overhead. We can implement a lightweight Set which is similar
> to {{LightWeightGSet}}, but not a fixed size ({{LightWeightGSet}} uses fix
> size for the entries array, usually it's a big value, an example is
> {{BlocksMap}}, this can avoid full gc since no need to resize), also we
> should be able to get Element through key.
> Following is comparison of memory footprint If we implement a lightweight set
> as described:
> We can save:
> {noformat}
> SIZE (bytes) ITEM
> 20The Key: Long (12 bytes object overhead + 8
> bytes long)
> 12HashMap Entry object overhead
> 4
[jira] [Commented] (HDFS-8805) Archival Storage: getStoragePolicy should not need superuser privilege
[
https://issues.apache.org/jira/browse/HDFS-8805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14693349#comment-14693349
]
Hudson commented on HDFS-8805:
--
FAILURE: Integrated in Hadoop-Yarn-trunk #1015 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/1015/])
HDFS-8805. Archival Storage: getStoragePolicy should not need superuser
privilege. Contributed by Brahma Reddy Battula. (jing9: rev
1fc3c779a422bafdb86ad1a5b2349802dda1cb62)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirAppendOp.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirWriteFileOp.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirStatAndListingOp.java
> Archival Storage: getStoragePolicy should not need superuser privilege
> --
>
> Key: HDFS-8805
> URL: https://issues.apache.org/jira/browse/HDFS-8805
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: balancer & mover, namenode
>Reporter: Hui Zheng
>Assignee: Brahma Reddy Battula
> Fix For: 2.8.0
>
> Attachments: HDFS-8805-002.patch, HDFS-8805-003.patch,
> HDFS-8805-004.patch, HDFS-8805.patch
>
>
> The result of getStoragePolicy command is always 'unspecified' even we has
> set a StoragePolicy on a directory.But the real placement of blocks is
> correct.
> The result of fsck is not correct either.
> {code}
> $ hdfs storagepolicies -setStoragePolicy -path /tmp/cold -policy COLD
> Set storage policy COLD on /tmp/cold
> $ hdfs storagepolicies -getStoragePolicy -path /tmp/cold
> The storage policy of /tmp/cold is unspecified
> $ hdfs fsck -storagepolicies /tmp/cold
> Blocks NOT satisfying the specified storage policy:
> Storage Policy Specified Storage Policy # of blocks
> % of blocks
> ARCHIVE:4(COLD) HOT 5
>55.5556%
> ARCHIVE:3(COLD) HOT 4
>44.%
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8887) Expose storage type and storage ID in BlockLocation
[ https://issues.apache.org/jira/browse/HDFS-8887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14693351#comment-14693351 ] Hudson commented on HDFS-8887: -- FAILURE: Integrated in Hadoop-Yarn-trunk #1015 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/1015/]) HDFS-8887. Expose storage type and storage ID in BlockLocation. (wang: rev 1ea1a8334ea01814121490a5bfd2a0205c66d6e4) * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/fs/BlockStorageLocation.java * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestBlockLocation.java * hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSUtilClient.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/BlockLocation.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java > Expose storage type and storage ID in BlockLocation > --- > > Key: HDFS-8887 > URL: https://issues.apache.org/jira/browse/HDFS-8887 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 2.7.1 >Reporter: Andrew Wang >Assignee: Andrew Wang > Fix For: 2.8.0 > > Attachments: HDFS-8887.001.patch, HDFS-8887.002.patch > > > Some applications schedule based on info like storage type or storage ID, > it'd be useful to expose this information in BlockLocation. It's already > included in LocatedBlock and sent over the wire. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8887) Expose storage type and storage ID in BlockLocation
[ https://issues.apache.org/jira/browse/HDFS-8887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14693355#comment-14693355 ] Hudson commented on HDFS-8887: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #285 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/285/]) HDFS-8887. Expose storage type and storage ID in BlockLocation. (wang: rev 1ea1a8334ea01814121490a5bfd2a0205c66d6e4) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/fs/BlockStorageLocation.java * hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSUtilClient.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/BlockLocation.java * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestBlockLocation.java > Expose storage type and storage ID in BlockLocation > --- > > Key: HDFS-8887 > URL: https://issues.apache.org/jira/browse/HDFS-8887 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 2.7.1 >Reporter: Andrew Wang >Assignee: Andrew Wang > Fix For: 2.8.0 > > Attachments: HDFS-8887.001.patch, HDFS-8887.002.patch > > > Some applications schedule based on info like storage type or storage ID, > it'd be useful to expose this information in BlockLocation. It's already > included in LocatedBlock and sent over the wire. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8805) Archival Storage: getStoragePolicy should not need superuser privilege
[
https://issues.apache.org/jira/browse/HDFS-8805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14693353#comment-14693353
]
Hudson commented on HDFS-8805:
--
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #285 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/285/])
HDFS-8805. Archival Storage: getStoragePolicy should not need superuser
privilege. Contributed by Brahma Reddy Battula. (jing9: rev
1fc3c779a422bafdb86ad1a5b2349802dda1cb62)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirStatAndListingOp.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirWriteFileOp.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirAppendOp.java
> Archival Storage: getStoragePolicy should not need superuser privilege
> --
>
> Key: HDFS-8805
> URL: https://issues.apache.org/jira/browse/HDFS-8805
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: balancer & mover, namenode
>Reporter: Hui Zheng
>Assignee: Brahma Reddy Battula
> Fix For: 2.8.0
>
> Attachments: HDFS-8805-002.patch, HDFS-8805-003.patch,
> HDFS-8805-004.patch, HDFS-8805.patch
>
>
> The result of getStoragePolicy command is always 'unspecified' even we has
> set a StoragePolicy on a directory.But the real placement of blocks is
> correct.
> The result of fsck is not correct either.
> {code}
> $ hdfs storagepolicies -setStoragePolicy -path /tmp/cold -policy COLD
> Set storage policy COLD on /tmp/cold
> $ hdfs storagepolicies -getStoragePolicy -path /tmp/cold
> The storage policy of /tmp/cold is unspecified
> $ hdfs fsck -storagepolicies /tmp/cold
> Blocks NOT satisfying the specified storage policy:
> Storage Policy Specified Storage Policy # of blocks
> % of blocks
> ARCHIVE:4(COLD) HOT 5
>55.5556%
> ARCHIVE:3(COLD) HOT 4
>44.%
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7899) Improve EOF error message
[
https://issues.apache.org/jira/browse/HDFS-7899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kiran Kumar M R updated HDFS-7899:
--
Assignee: (was: Kiran Kumar M R)
> Improve EOF error message
> -
>
> Key: HDFS-7899
> URL: https://issues.apache.org/jira/browse/HDFS-7899
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Affects Versions: 2.6.0
>Reporter: Harsh J
>Priority: Minor
>
> Currently, a DN disconnection for reasons other than connection timeout or
> refused messages, such as an EOF message as a result of rejection or other
> network fault, reports in this manner:
> {code}
> WARN org.apache.hadoop.hdfs.DFSClient: Failed to connect to /x.x.x.x: for
> block, add to deadNodes and continue. java.io.EOFException: Premature EOF: no
> length prefix available
> java.io.EOFException: Premature EOF: no length prefix available
> at
> org.apache.hadoop.hdfs.protocol.HdfsProtoUtil.vintPrefixed(HdfsProtoUtil.java:171)
>
> at
> org.apache.hadoop.hdfs.RemoteBlockReader2.newBlockReader(RemoteBlockReader2.java:392)
>
> at
> org.apache.hadoop.hdfs.BlockReaderFactory.newBlockReader(BlockReaderFactory.java:137)
>
> at
> org.apache.hadoop.hdfs.DFSInputStream.getBlockReader(DFSInputStream.java:1103)
>
> at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:538)
> at
> org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:750)
>
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:794)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:602)
> {code}
> This is not very clear to a user (warn's at the hdfs-client). It could likely
> be improved with a more diagnosable message, or at least the direct reason
> than an EOF.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8312) Trash does not descent into child directories to check for permissions
[
https://issues.apache.org/jira/browse/HDFS-8312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kiran Kumar M R updated HDFS-8312:
--
Assignee: (was: Kiran Kumar M R)
> Trash does not descent into child directories to check for permissions
> --
>
> Key: HDFS-8312
> URL: https://issues.apache.org/jira/browse/HDFS-8312
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: HDFS, security
>Affects Versions: 2.2.0, 2.6.0
>Reporter: Eric Yang
>
> HDFS trash does not descent into child directory to check if user has
> permission to delete files. For example:
> Run the following command to initialize directory structure as super user:
> {code}
> hadoop fs -mkdir /BSS/level1
> hadoop fs -mkdir /BSS/level1/level2
> hadoop fs -mkdir /BSS/level1/level2/level3
> hadoop fs -put /tmp/appConfig.json /BSS/level1/level2/level3/testfile.txt
> hadoop fs -chown user1:users /BSS/level1/level2/level3/testfile.txt
> hadoop fs -chown -R user1:users /BSS/level1
> hadoop fs -chown -R 750 /BSS/level1
> hadoop fs -chmod -R 640 /BSS/level1/level2/level3/testfile.txt
> hadoop fs -chmod 775 /BSS
> {code}
> Change to a normal user called user2.
> When trash is enabled:
> {code}
> sudo su user2 -
> hadoop fs -rm -r /BSS/level1
> 15/05/01 16:51:20 INFO fs.TrashPolicyDefault: Namenode trash configuration:
> Deletion interval = 3600 minutes, Emptier interval = 0 minutes.
> Moved: 'hdfs://bdvs323.svl.ibm.com:9000/BSS/level1' to trash at:
> hdfs://bdvs323.svl.ibm.com:9000/user/user2/.Trash/Current
> {code}
> When trash is disabled:
> {code}
> /opt/ibm/biginsights/IHC/bin/hadoop fs -Dfs.trash.interval=0 -rm -r
> /BSS/level1
> 15/05/01 16:58:31 INFO fs.TrashPolicyDefault: Namenode trash configuration:
> Deletion interval = 0 minutes, Emptier interval = 0 minutes.
> rm: Permission denied: user=user2, access=ALL,
> inode="/BSS/level1":user1:users:drwxr-x---
> {code}
> There is inconsistency between trash behavior and delete behavior. When
> trash is enabled, files owned by user1 is deleted by user2. It looks like
> trash does not recursively validate if the child directory files can be
> removed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7492) If multiple threads call FsVolumeList#checkDirs at the same time, we should only do checkDirs once and give the results to all waiting threads
[ https://issues.apache.org/jira/browse/HDFS-7492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kiran Kumar M R updated HDFS-7492: -- Assignee: (was: Kiran Kumar M R) > If multiple threads call FsVolumeList#checkDirs at the same time, we should > only do checkDirs once and give the results to all waiting threads > -- > > Key: HDFS-7492 > URL: https://issues.apache.org/jira/browse/HDFS-7492 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Colin Patrick McCabe >Priority: Minor > > checkDirs is called when we encounter certain I/O errors. It's rare to get > just a single I/O error... normally you start getting many errors when a disk > is going bad. For this reason, we shouldn't start a new checkDirs scan for > each error. Instead, if multiple threads call FsVolumeList#checkDirs at > around the same time, we should only do checkDirs once and give the results > to all the waiting threads. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8887) Expose storage type and storage ID in BlockLocation
[ https://issues.apache.org/jira/browse/HDFS-8887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14693478#comment-14693478 ] Hudson commented on HDFS-8887: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #2212 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2212/]) HDFS-8887. Expose storage type and storage ID in BlockLocation. (wang: rev 1ea1a8334ea01814121490a5bfd2a0205c66d6e4) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestBlockLocation.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java * hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSUtilClient.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/BlockLocation.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/fs/BlockStorageLocation.java > Expose storage type and storage ID in BlockLocation > --- > > Key: HDFS-8887 > URL: https://issues.apache.org/jira/browse/HDFS-8887 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 2.7.1 >Reporter: Andrew Wang >Assignee: Andrew Wang > Fix For: 2.8.0 > > Attachments: HDFS-8887.001.patch, HDFS-8887.002.patch > > > Some applications schedule based on info like storage type or storage ID, > it'd be useful to expose this information in BlockLocation. It's already > included in LocatedBlock and sent over the wire. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8805) Archival Storage: getStoragePolicy should not need superuser privilege
[
https://issues.apache.org/jira/browse/HDFS-8805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14693476#comment-14693476
]
Hudson commented on HDFS-8805:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk #2212 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/2212/])
HDFS-8805. Archival Storage: getStoragePolicy should not need superuser
privilege. Contributed by Brahma Reddy Battula. (jing9: rev
1fc3c779a422bafdb86ad1a5b2349802dda1cb62)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirWriteFileOp.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirAppendOp.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirStatAndListingOp.java
> Archival Storage: getStoragePolicy should not need superuser privilege
> --
>
> Key: HDFS-8805
> URL: https://issues.apache.org/jira/browse/HDFS-8805
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: balancer & mover, namenode
>Reporter: Hui Zheng
>Assignee: Brahma Reddy Battula
> Fix For: 2.8.0
>
> Attachments: HDFS-8805-002.patch, HDFS-8805-003.patch,
> HDFS-8805-004.patch, HDFS-8805.patch
>
>
> The result of getStoragePolicy command is always 'unspecified' even we has
> set a StoragePolicy on a directory.But the real placement of blocks is
> correct.
> The result of fsck is not correct either.
> {code}
> $ hdfs storagepolicies -setStoragePolicy -path /tmp/cold -policy COLD
> Set storage policy COLD on /tmp/cold
> $ hdfs storagepolicies -getStoragePolicy -path /tmp/cold
> The storage policy of /tmp/cold is unspecified
> $ hdfs fsck -storagepolicies /tmp/cold
> Blocks NOT satisfying the specified storage policy:
> Storage Policy Specified Storage Policy # of blocks
> % of blocks
> ARCHIVE:4(COLD) HOT 5
>55.5556%
> ARCHIVE:3(COLD) HOT 4
>44.%
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8622) Implement GETCONTENTSUMMARY operation for WebImageViewer
[
https://issues.apache.org/jira/browse/HDFS-8622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jagadesh Kiran N updated HDFS-8622:
---
Attachment: HDFS-8622-10.patch
Hi [~ajisakaa] updated the review comment. Pls check
> Implement GETCONTENTSUMMARY operation for WebImageViewer
>
>
> Key: HDFS-8622
> URL: https://issues.apache.org/jira/browse/HDFS-8622
> Project: Hadoop HDFS
> Issue Type: New Feature
>Reporter: Jagadesh Kiran N
>Assignee: Jagadesh Kiran N
> Attachments: HDFS-8622-00.patch, HDFS-8622-01.patch,
> HDFS-8622-02.patch, HDFS-8622-03.patch, HDFS-8622-04.patch,
> HDFS-8622-05.patch, HDFS-8622-06.patch, HDFS-8622-07.patch,
> HDFS-8622-08.patch, HDFS-8622-09.patch, HDFS-8622-10.patch
>
>
> it would be better for administrators if {code} GETCONTENTSUMMARY {code} are
> supported.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8805) Archival Storage: getStoragePolicy should not need superuser privilege
[
https://issues.apache.org/jira/browse/HDFS-8805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14693574#comment-14693574
]
Hudson commented on HDFS-8805:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #282 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/282/])
HDFS-8805. Archival Storage: getStoragePolicy should not need superuser
privilege. Contributed by Brahma Reddy Battula. (jing9: rev
1fc3c779a422bafdb86ad1a5b2349802dda1cb62)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirWriteFileOp.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirAppendOp.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirStatAndListingOp.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
> Archival Storage: getStoragePolicy should not need superuser privilege
> --
>
> Key: HDFS-8805
> URL: https://issues.apache.org/jira/browse/HDFS-8805
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: balancer & mover, namenode
>Reporter: Hui Zheng
>Assignee: Brahma Reddy Battula
> Fix For: 2.8.0
>
> Attachments: HDFS-8805-002.patch, HDFS-8805-003.patch,
> HDFS-8805-004.patch, HDFS-8805.patch
>
>
> The result of getStoragePolicy command is always 'unspecified' even we has
> set a StoragePolicy on a directory.But the real placement of blocks is
> correct.
> The result of fsck is not correct either.
> {code}
> $ hdfs storagepolicies -setStoragePolicy -path /tmp/cold -policy COLD
> Set storage policy COLD on /tmp/cold
> $ hdfs storagepolicies -getStoragePolicy -path /tmp/cold
> The storage policy of /tmp/cold is unspecified
> $ hdfs fsck -storagepolicies /tmp/cold
> Blocks NOT satisfying the specified storage policy:
> Storage Policy Specified Storage Policy # of blocks
> % of blocks
> ARCHIVE:4(COLD) HOT 5
>55.5556%
> ARCHIVE:3(COLD) HOT 4
>44.%
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8887) Expose storage type and storage ID in BlockLocation
[ https://issues.apache.org/jira/browse/HDFS-8887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14693576#comment-14693576 ] Hudson commented on HDFS-8887: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #282 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/282/]) HDFS-8887. Expose storage type and storage ID in BlockLocation. (wang: rev 1ea1a8334ea01814121490a5bfd2a0205c66d6e4) * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSUtilClient.java * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestBlockLocation.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/BlockLocation.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/fs/BlockStorageLocation.java > Expose storage type and storage ID in BlockLocation > --- > > Key: HDFS-8887 > URL: https://issues.apache.org/jira/browse/HDFS-8887 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 2.7.1 >Reporter: Andrew Wang >Assignee: Andrew Wang > Fix For: 2.8.0 > > Attachments: HDFS-8887.001.patch, HDFS-8887.002.patch > > > Some applications schedule based on info like storage type or storage ID, > it'd be useful to expose this information in BlockLocation. It's already > included in LocatedBlock and sent over the wire. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8887) Expose storage type and storage ID in BlockLocation
[ https://issues.apache.org/jira/browse/HDFS-8887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14693588#comment-14693588 ] Hudson commented on HDFS-8887: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #2231 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2231/]) HDFS-8887. Expose storage type and storage ID in BlockLocation. (wang: rev 1ea1a8334ea01814121490a5bfd2a0205c66d6e4) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/fs/BlockStorageLocation.java * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestBlockLocation.java * hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSUtilClient.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/BlockLocation.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java > Expose storage type and storage ID in BlockLocation > --- > > Key: HDFS-8887 > URL: https://issues.apache.org/jira/browse/HDFS-8887 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 2.7.1 >Reporter: Andrew Wang >Assignee: Andrew Wang > Fix For: 2.8.0 > > Attachments: HDFS-8887.001.patch, HDFS-8887.002.patch > > > Some applications schedule based on info like storage type or storage ID, > it'd be useful to expose this information in BlockLocation. It's already > included in LocatedBlock and sent over the wire. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8805) Archival Storage: getStoragePolicy should not need superuser privilege
[
https://issues.apache.org/jira/browse/HDFS-8805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14693586#comment-14693586
]
Hudson commented on HDFS-8805:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk #2231 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2231/])
HDFS-8805. Archival Storage: getStoragePolicy should not need superuser
privilege. Contributed by Brahma Reddy Battula. (jing9: rev
1fc3c779a422bafdb86ad1a5b2349802dda1cb62)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirStatAndListingOp.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirAppendOp.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirWriteFileOp.java
> Archival Storage: getStoragePolicy should not need superuser privilege
> --
>
> Key: HDFS-8805
> URL: https://issues.apache.org/jira/browse/HDFS-8805
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: balancer & mover, namenode
>Reporter: Hui Zheng
>Assignee: Brahma Reddy Battula
> Fix For: 2.8.0
>
> Attachments: HDFS-8805-002.patch, HDFS-8805-003.patch,
> HDFS-8805-004.patch, HDFS-8805.patch
>
>
> The result of getStoragePolicy command is always 'unspecified' even we has
> set a StoragePolicy on a directory.But the real placement of blocks is
> correct.
> The result of fsck is not correct either.
> {code}
> $ hdfs storagepolicies -setStoragePolicy -path /tmp/cold -policy COLD
> Set storage policy COLD on /tmp/cold
> $ hdfs storagepolicies -getStoragePolicy -path /tmp/cold
> The storage policy of /tmp/cold is unspecified
> $ hdfs fsck -storagepolicies /tmp/cold
> Blocks NOT satisfying the specified storage policy:
> Storage Policy Specified Storage Policy # of blocks
> % of blocks
> ARCHIVE:4(COLD) HOT 5
>55.5556%
> ARCHIVE:3(COLD) HOT 4
>44.%
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8859) Improve DataNode (ReplicaMap) memory footprint to save about 45%
[
https://issues.apache.org/jira/browse/HDFS-8859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14693629#comment-14693629
]
Yi Liu commented on HDFS-8859:
--
{{TestRestartDFS}} is related to {{003}} patch, but {{002}} doesn't cause any
issue. I just debug it, the reason seems the original implementation of
{{SetIterator}} in {{LightWeightGSet}} has some issue, I wrote a more clear
{{SetIterator}} in the new class {{LightWeightHashGSet}} in {{002}}, but in
{{003}}, I make it to extend {{LightWeightGSet}} but not use my new
implementation of {{SetIterator}}. If I use my new implementation of
{{SetIterator}}, then the failure disappears.
Let me find some time later to see why original implementation of
{{SetIterator}} in {{LightWeightGSet}} causes the failure (it was not used in
original code, so the bug might not be found).
> Improve DataNode (ReplicaMap) memory footprint to save about 45%
>
>
> Key: HDFS-8859
> URL: https://issues.apache.org/jira/browse/HDFS-8859
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Reporter: Yi Liu
>Assignee: Yi Liu
>Priority: Critical
> Attachments: HDFS-8859.001.patch, HDFS-8859.002.patch,
> HDFS-8859.003.patch
>
>
> By using following approach we can save about *45%* memory footprint for each
> block replica in DataNode memory (This JIRA only talks about *ReplicaMap* in
> DataNode), the details are:
> In ReplicaMap,
> {code}
> private final Map> map =
> new HashMap>();
> {code}
> Currently we use a HashMap {{Map}} to store the replicas
> in memory. The key is block id of the block replica which is already
> included in {{ReplicaInfo}}, so this memory can be saved. Also HashMap Entry
> has a object overhead. We can implement a lightweight Set which is similar
> to {{LightWeightGSet}}, but not a fixed size ({{LightWeightGSet}} uses fix
> size for the entries array, usually it's a big value, an example is
> {{BlocksMap}}, this can avoid full gc since no need to resize), also we
> should be able to get Element through key.
> Following is comparison of memory footprint If we implement a lightweight set
> as described:
> We can save:
> {noformat}
> SIZE (bytes) ITEM
> 20The Key: Long (12 bytes object overhead + 8
> bytes long)
> 12HashMap Entry object overhead
> 4 reference to the key in Entry
> 4 reference to the value in Entry
> 4 hash in Entry
> {noformat}
> Total: -44 bytes
> We need to add:
> {noformat}
> SIZE (bytes) ITEM
> 4 a reference to next element in ReplicaInfo
> {noformat}
> Total: +4 bytes
> So totally we can save 40bytes for each block replica
> And currently one finalized replica needs around 46 bytes (notice: we ignore
> memory alignment here).
> We can save 1 - (4 + 46) / (44 + 46) = *45%* memory for each block replica
> in DataNode.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8805) Archival Storage: getStoragePolicy should not need superuser privilege
[
https://issues.apache.org/jira/browse/HDFS-8805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14693655#comment-14693655
]
Hudson commented on HDFS-8805:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #274 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/274/])
HDFS-8805. Archival Storage: getStoragePolicy should not need superuser
privilege. Contributed by Brahma Reddy Battula. (jing9: rev
1fc3c779a422bafdb86ad1a5b2349802dda1cb62)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirStatAndListingOp.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirWriteFileOp.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirAppendOp.java
> Archival Storage: getStoragePolicy should not need superuser privilege
> --
>
> Key: HDFS-8805
> URL: https://issues.apache.org/jira/browse/HDFS-8805
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: balancer & mover, namenode
>Reporter: Hui Zheng
>Assignee: Brahma Reddy Battula
> Fix For: 2.8.0
>
> Attachments: HDFS-8805-002.patch, HDFS-8805-003.patch,
> HDFS-8805-004.patch, HDFS-8805.patch
>
>
> The result of getStoragePolicy command is always 'unspecified' even we has
> set a StoragePolicy on a directory.But the real placement of blocks is
> correct.
> The result of fsck is not correct either.
> {code}
> $ hdfs storagepolicies -setStoragePolicy -path /tmp/cold -policy COLD
> Set storage policy COLD on /tmp/cold
> $ hdfs storagepolicies -getStoragePolicy -path /tmp/cold
> The storage policy of /tmp/cold is unspecified
> $ hdfs fsck -storagepolicies /tmp/cold
> Blocks NOT satisfying the specified storage policy:
> Storage Policy Specified Storage Policy # of blocks
> % of blocks
> ARCHIVE:4(COLD) HOT 5
>55.5556%
> ARCHIVE:3(COLD) HOT 4
>44.%
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8887) Expose storage type and storage ID in BlockLocation
[ https://issues.apache.org/jira/browse/HDFS-8887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14693657#comment-14693657 ] Hudson commented on HDFS-8887: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #274 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/274/]) HDFS-8887. Expose storage type and storage ID in BlockLocation. (wang: rev 1ea1a8334ea01814121490a5bfd2a0205c66d6e4) * hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSUtilClient.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/fs/BlockStorageLocation.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/BlockLocation.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestBlockLocation.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java > Expose storage type and storage ID in BlockLocation > --- > > Key: HDFS-8887 > URL: https://issues.apache.org/jira/browse/HDFS-8887 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 2.7.1 >Reporter: Andrew Wang >Assignee: Andrew Wang > Fix For: 2.8.0 > > Attachments: HDFS-8887.001.patch, HDFS-8887.002.patch > > > Some applications schedule based on info like storage type or storage ID, > it'd be useful to expose this information in BlockLocation. It's already > included in LocatedBlock and sent over the wire. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8078) HDFS client gets errors trying to to connect to IPv6 DataNode
[ https://issues.apache.org/jira/browse/HDFS-8078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14693681#comment-14693681 ] Allen Wittenauer commented on HDFS-8078: [~nkedel]: please take a second to check which box you are editing when submitting. You've updated the release note field several times instead of adding a comment. :) > HDFS client gets errors trying to to connect to IPv6 DataNode > - > > Key: HDFS-8078 > URL: https://issues.apache.org/jira/browse/HDFS-8078 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Affects Versions: 2.6.0 >Reporter: Nate Edel >Assignee: Nate Edel > Labels: BB2015-05-TBR, ipv6 > Attachments: HDFS-8078.10.patch, HDFS-8078.11.patch, > HDFS-8078.12.patch, HDFS-8078.13.patch, HDFS-8078.14.patch, HDFS-8078.9.patch > > > 1st exception, on put: > 15/03/23 18:43:18 WARN hdfs.DFSClient: DataStreamer Exception > java.lang.IllegalArgumentException: Does not contain a valid host:port > authority: 2401:db00:1010:70ba:face:0:8:0:50010 > at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:212) > at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:164) > at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:153) > at > org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1607) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1408) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1361) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:588) > Appears to actually stem from code in DataNodeID which assumes it's safe to > append together (ipaddr + ":" + port) -- which is OK for IPv4 and not OK for > IPv6. NetUtils.createSocketAddr( ) assembles a Java URI object, which > requires the format proto://[2401:db00:1010:70ba:face:0:8:0]:50010 > Currently using InetAddress.getByName() to validate IPv6 (guava > InetAddresses.forString has been flaky) but could also use our own parsing. > (From logging this, it seems like a low-enough frequency call that the extra > object creation shouldn't be problematic, and for me the slight risk of > passing in bad input that is not actually an IPv4 or IPv6 address and thus > calling an external DNS lookup is outweighed by getting the address > normalized and avoiding rewriting parsing.) > Alternatively, sun.net.util.IPAddressUtil.isIPv6LiteralAddress() > --- > 2nd exception (on datanode) > 15/04/13 13:18:07 ERROR datanode.DataNode: > dev1903.prn1.facebook.com:50010:DataXceiver error processing unknown > operation src: /2401:db00:20:7013:face:0:7:0:54152 dst: > /2401:db00:11:d010:face:0:2f:0:50010 > java.io.EOFException > at java.io.DataInputStream.readShort(DataInputStream.java:315) > at > org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:226) > at java.lang.Thread.run(Thread.java:745) > Which also comes as client error "-get: 2401 is not an IP string literal." > This one has existing parsing logic which needs to shift to the last colon > rather than the first. Should also be a tiny bit faster by using lastIndexOf > rather than split. Could alternatively use the techniques above. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8859) Improve DataNode (ReplicaMap) memory footprint to save about 45%
[
https://issues.apache.org/jira/browse/HDFS-8859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14693857#comment-14693857
]
Tsz Wo Nicholas Sze commented on HDFS-8859:
---
The idea sound good. Some comments:
- Both LightWeightGSet and the new LightWeightHashGSet use hash functions. So
LightWeightHashGSet seems not a good name. How about calling it
LightWeightResizableGSet?
- From your calculation, the patch improve each block replica object size about
45%. The JIRA summary is misleading. It seems claiming that it improves the
overall DataNode memory footprint by about 45%. For 10m replicas, the original
overall map entry object size is ~900 MB and the new size is ~500MB. Is it
correct?
- Why adding LightWeightGSet.putElement? Subclass can call super.put(..).
- There is a rewrite for LightWeightGSet.remove(..). Why? The old code is
well tested. Please do not change it if possible.
- Took a quick looks at the tests. I think we need some long running tests to
make sure the correctness. See TestGSet.runMultipleTestGSet().
> Improve DataNode (ReplicaMap) memory footprint to save about 45%
>
>
> Key: HDFS-8859
> URL: https://issues.apache.org/jira/browse/HDFS-8859
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Reporter: Yi Liu
>Assignee: Yi Liu
>Priority: Critical
> Attachments: HDFS-8859.001.patch, HDFS-8859.002.patch,
> HDFS-8859.003.patch
>
>
> By using following approach we can save about *45%* memory footprint for each
> block replica in DataNode memory (This JIRA only talks about *ReplicaMap* in
> DataNode), the details are:
> In ReplicaMap,
> {code}
> private final Map> map =
> new HashMap>();
> {code}
> Currently we use a HashMap {{Map}} to store the replicas
> in memory. The key is block id of the block replica which is already
> included in {{ReplicaInfo}}, so this memory can be saved. Also HashMap Entry
> has a object overhead. We can implement a lightweight Set which is similar
> to {{LightWeightGSet}}, but not a fixed size ({{LightWeightGSet}} uses fix
> size for the entries array, usually it's a big value, an example is
> {{BlocksMap}}, this can avoid full gc since no need to resize), also we
> should be able to get Element through key.
> Following is comparison of memory footprint If we implement a lightweight set
> as described:
> We can save:
> {noformat}
> SIZE (bytes) ITEM
> 20The Key: Long (12 bytes object overhead + 8
> bytes long)
> 12HashMap Entry object overhead
> 4 reference to the key in Entry
> 4 reference to the value in Entry
> 4 hash in Entry
> {noformat}
> Total: -44 bytes
> We need to add:
> {noformat}
> SIZE (bytes) ITEM
> 4 a reference to next element in ReplicaInfo
> {noformat}
> Total: +4 bytes
> So totally we can save 40bytes for each block replica
> And currently one finalized replica needs around 46 bytes (notice: we ignore
> memory alignment here).
> We can save 1 - (4 + 46) / (44 + 46) = *45%* memory for each block replica
> in DataNode.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8388) Time and Date format need to be in sync in Namenode UI page
[
https://issues.apache.org/jira/browse/HDFS-8388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14693887#comment-14693887
]
Surendra Singh Lilhore commented on HDFS-8388:
--
Thanks [~ajisakaa] for comments..
I didn't see any document for {{NNStarted}}, so I didn't added for new metric.
http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/Metrics.html
> Time and Date format need to be in sync in Namenode UI page
> ---
>
> Key: HDFS-8388
> URL: https://issues.apache.org/jira/browse/HDFS-8388
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Archana T
>Assignee: Surendra Singh Lilhore
>Priority: Minor
> Attachments: HDFS-8388-002.patch, HDFS-8388-003.patch,
> HDFS-8388.patch, HDFS-8388_1.patch, ScreenShot-InvalidDate.png
>
>
> In NameNode UI Page, Date and Time FORMAT displayed on the page are not in
> sync currently.
> Started:Wed May 13 12:28:02 IST 2015
> Compiled:23 Apr 2015 12:22:59
> Block Deletion Start Time 13 May 2015 12:28:02
> We can keep a common format in all the above places.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8883) NameNode Metrics : Add FSNameSystem lock Queue Length
[ https://issues.apache.org/jira/browse/HDFS-8883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDFS-8883: --- Status: Patch Available (was: Open) > NameNode Metrics : Add FSNameSystem lock Queue Length > - > > Key: HDFS-8883 > URL: https://issues.apache.org/jira/browse/HDFS-8883 > Project: Hadoop HDFS > Issue Type: Improvement > Components: HDFS >Affects Versions: 2.7.1 >Reporter: Anu Engineer >Assignee: Anu Engineer >Priority: Minor > Fix For: 2.8.0 > > Attachments: HDFS-8883.001.patch > > > FSNameSystemLock can have contention when NameNode is under load. This patch > adds LockQueueLength -- the number of threads waiting on FSNameSystemLock -- > as a metric in NameNode. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8833) Erasure coding: store EC schema and cell size in INodeFile and eliminate notion of EC zones
[
https://issues.apache.org/jira/browse/HDFS-8833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14693958#comment-14693958
]
Zhe Zhang commented on HDFS-8833:
-
Thanks Jing for the helpful feedback.
bq. Will we allow associating the EC policy with a non-empty directory? I guess
we should disallow it, otherwise the semantic of the "create EC Directory"
command can be very confusing.
With file renaming allowed, I don't think users should expect the EC policy of
a directory to represent policies of its files anyway, right? I think the EC
policy of a directory should only affect new file creation, a similar semantics
as the storage policy on a directory.
Sorry that the proposed [CLI change |
https://issues.apache.org/jira/browse/HDFS-8833?focusedCommentId=14647251&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14647251]
is a little outdated. As Vinay [suggested |
https://issues.apache.org/jira/browse/HDFS-8833?focusedCommentId=14647360&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14647360],
I plan to allow {{setErasureCodingPolicy}} on non-empty directories. I guess
"create EC Directory" in your comment refers to this command?
bq. Do we want to allow nested EC directories? Currently since we only support
one policy, I do not see any benefits to have nested EC directories.
That's a good point. With 1 EC policy, I agree we should defer this decision
until we have at least 2 policies.
> Erasure coding: store EC schema and cell size in INodeFile and eliminate
> notion of EC zones
> ---
>
> Key: HDFS-8833
> URL: https://issues.apache.org/jira/browse/HDFS-8833
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: HDFS-7285
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
>
> We have [discussed |
> https://issues.apache.org/jira/browse/HDFS-7285?focusedCommentId=14357754&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14357754]
> storing EC schema with files instead of EC zones and recently revisited the
> discussion under HDFS-8059.
> As a recap, the _zone_ concept has severe limitations including renaming and
> nested configuration. Those limitations are valid in encryption for security
> reasons and it doesn't make sense to carry them over in EC.
> This JIRA aims to store EC schema and cell size on {{INodeFile}} level. For
> simplicity, we should first implement it as an xattr and consider memory
> optimizations (such as moving it to file header) as a follow-on. We should
> also disable changing EC policy on a non-empty file / dir in the first phase.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8622) Implement GETCONTENTSUMMARY operation for WebImageViewer
[
https://issues.apache.org/jira/browse/HDFS-8622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14693961#comment-14693961
]
Hadoop QA commented on HDFS-8622:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 21m 4s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 49s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 44s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | site | 2m 59s | Site still builds. |
| {color:green}+1{color} | checkstyle | 1m 26s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 23s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 34s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 3s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 176m 38s | Tests failed in hadoop-hdfs. |
| | | 227m 40s | |
\\
\\
|| Reason || Tests ||
| Timed out tests | org.apache.hadoop.cli.TestHDFSCLI |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12750079/HDFS-8622-10.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle site |
| git revision | trunk / 1c12adb |
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/11981/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/11981/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/11981/console |
This message was automatically generated.
> Implement GETCONTENTSUMMARY operation for WebImageViewer
>
>
> Key: HDFS-8622
> URL: https://issues.apache.org/jira/browse/HDFS-8622
> Project: Hadoop HDFS
> Issue Type: New Feature
>Reporter: Jagadesh Kiran N
>Assignee: Jagadesh Kiran N
> Attachments: HDFS-8622-00.patch, HDFS-8622-01.patch,
> HDFS-8622-02.patch, HDFS-8622-03.patch, HDFS-8622-04.patch,
> HDFS-8622-05.patch, HDFS-8622-06.patch, HDFS-8622-07.patch,
> HDFS-8622-08.patch, HDFS-8622-09.patch, HDFS-8622-10.patch
>
>
> it would be better for administrators if {code} GETCONTENTSUMMARY {code} are
> supported.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8854) Erasure coding: add ECPolicy to replace schema+cellSize in hadoop-hdfs
[
https://issues.apache.org/jira/browse/HDFS-8854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14693973#comment-14693973
]
Zhe Zhang commented on HDFS-8854:
-
Thanks Rakesh for pointing this out. Since this is a large patch, I pushed it
to my personal github repo and started a Jenkins [job |
https://builds.apache.org/job/Hadoop-HDFS-7285-Merge/].
[~aw] Could you advice how we can trigger Jenkins on the {{HDFS-7285-merge}}
branch in this case? Also, when attaching consolidated patches on HDFS-7285,
how can we trigger Jenkins to apply them on trunk instead of {{HDFS-7285}}
branch? Thanks much..
> Erasure coding: add ECPolicy to replace schema+cellSize in hadoop-hdfs
> --
>
> Key: HDFS-8854
> URL: https://issues.apache.org/jira/browse/HDFS-8854
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: HDFS-7285
>Reporter: Walter Su
>Assignee: Walter Su
> Attachments: HDFS-8854-Consolidated-20150806.02.txt,
> HDFS-8854-HDFS-7285-merge.03.patch, HDFS-8854-HDFS-7285-merge.03.txt,
> HDFS-8854-HDFS-7285.00.patch, HDFS-8854-HDFS-7285.01.patch,
> HDFS-8854-HDFS-7285.02.patch, HDFS-8854-HDFS-7285.03.patch, HDFS-8854.00.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8854) Erasure coding: add ECPolicy to replace schema+cellSize in hadoop-hdfs
[ https://issues.apache.org/jira/browse/HDFS-8854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14694000#comment-14694000 ] Allen Wittenauer commented on HDFS-8854: Caveat: this comment applies to the version of test-patch in trunk. It does not apply to the version in the yetus branch (HADOOP-12111). bq. Could you advice how we can trigger Jenkins on the HDFS-7285-merge branch in this case? You can't. test-patch doesn't support multiple dashes in branch names. bq. Also, when attaching consolidated patches on HDFS-7285, how can we trigger Jenkins to apply them on trunk instead of HDFS-7285 branch? To run a patch on trunk, don't put a branch name in the patch. > Erasure coding: add ECPolicy to replace schema+cellSize in hadoop-hdfs > -- > > Key: HDFS-8854 > URL: https://issues.apache.org/jira/browse/HDFS-8854 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7285 >Reporter: Walter Su >Assignee: Walter Su > Attachments: HDFS-8854-Consolidated-20150806.02.txt, > HDFS-8854-HDFS-7285-merge.03.patch, HDFS-8854-HDFS-7285-merge.03.txt, > HDFS-8854-HDFS-7285.00.patch, HDFS-8854-HDFS-7285.01.patch, > HDFS-8854-HDFS-7285.02.patch, HDFS-8854-HDFS-7285.03.patch, HDFS-8854.00.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8854) Erasure coding: add ECPolicy to replace schema+cellSize in hadoop-hdfs
[ https://issues.apache.org/jira/browse/HDFS-8854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14694009#comment-14694009 ] Zhe Zhang commented on HDFS-8854: - Thanks Allen for the comment. We did try different names for the HDFS-7285 consolidated patch. The latest [attempt | https://issues.apache.org/jira/browse/HDFS-7285?focusedCommentId=14680785&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14680785]. > Erasure coding: add ECPolicy to replace schema+cellSize in hadoop-hdfs > -- > > Key: HDFS-8854 > URL: https://issues.apache.org/jira/browse/HDFS-8854 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7285 >Reporter: Walter Su >Assignee: Walter Su > Attachments: HDFS-8854-Consolidated-20150806.02.txt, > HDFS-8854-HDFS-7285-merge.03.patch, HDFS-8854-HDFS-7285-merge.03.txt, > HDFS-8854-HDFS-7285.00.patch, HDFS-8854-HDFS-7285.01.patch, > HDFS-8854-HDFS-7285.02.patch, HDFS-8854-HDFS-7285.03.patch, HDFS-8854.00.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8775) SASL support for data transfer protocol in libhdfspp
[
https://issues.apache.org/jira/browse/HDFS-8775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14694015#comment-14694015
]
Haohui Mai commented on HDFS-8775:
--
Thanks for the reviews.
The code basically blindly follows the RFC thus I see little value of comments
that document the workflow of the protocol.
bq. What is the TEST_mock_cnonce for? Do we have a mechanism to strip it out of
release code? Can we #ifdef it out?
Used in the unit test. LevelDB does basically the same. It's better to keep
them as is to simplify the deployment and testing.
bq. Why is kMaxBufferSize 64k? Does that relate to some other constant (in
which case, can we import the symbol?) or is it just "64k should be enough for
anybody"?
Typical requests won't exceed 64k. Larger request should be considered
malformed and be rejected.
bq. GenerateCNonce(): is RAND_pseudo good enough for security in this case, or
should we be using (transitively) /dev/random?
In practice a pseudo random number is sufficient for the purpose of nonce.
bq. malform requests.
These requests are not necessarily malformed according to the RFC as long as
the required fields are presented.
bq. ParseFirstChallenge(): requires a "nonce" field in the message, but doesn't
use it
It does.
{code}
+ nonce_ = props["nonce"];
...
+ << ",nonce=\"" << QuoteString(nonce_) << "\""
{code}
bq. GetMD5Digest(): we should check the return values of the OpenSSL calls
It doesn't seem to be necessarily if the input is from memory. Here is a code
snippet from boringssl.
{code}
uint8_t *MD5(const uint8_t *data, size_t len, uint8_t *out) {
MD5_CTX ctx;
static uint8_t digest[MD5_DIGEST_LENGTH];
/* TODO(fork): remove this static buffer. */
if (out == NULL) {
out = digest;
}
MD5_Init(&ctx);
MD5_Update(&ctx, data, len);
MD5_Final(out, &ctx);
return out;
}
{code}
bq. Negative tests
They would be nice to have but maybe we can add them in separate jiras.
> SASL support for data transfer protocol in libhdfspp
>
>
> Key: HDFS-8775
> URL: https://issues.apache.org/jira/browse/HDFS-8775
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Attachments: HDFS-8775.000.patch
>
>
> This jira proposes to implement basic SASL support for the data transfer
> protocol which allows libhdfspp to talk to secure clusters.
> Support for encryption is deferred to subsequent jiras.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8855) Webhdfs client leaks active NameNode connections
[
https://issues.apache.org/jira/browse/HDFS-8855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14694104#comment-14694104
]
Xiaobing Zhou commented on HDFS-8855:
-
[~bobhansen]
I was using the sequential work load for results above.
{noformat}
#!/bin/bash
# segment, op=OPEN and offset are added to url_base
count=${count:-100}
#echo $count
url_base=${url_base:-"http://c6401.ambari.apache.org:50070/webhdfs/v1/tmp/bigfile"}
#echo $url_base
read_size=${read_size:-1}
#echo $read_size
file_size=${file_size:-$[ 1024 * 1024 * 1024 ]}
#echo $file_size
#namenode=${namenode:-`echo $url_base | grep -Po "(?<=http://)[^:/]*"`}
namenode="c6401.ambari.apache.org"
#echo $namenode
for i in `seq 1 $count` ; do
rand=$(od -N 4 -t uL -An /dev/urandom | tr -d " ")
#echo $rand
offset=$[ ( $rand % (file_size / read_size) * read_size )]
#echo $offset
url=$url_base?op=OPEN\&offset=$offset\&length=$read_size
#echo $url
curl -L "$url" > url.blah 2>/dev/null
#curl -L "$url"
if (( $i % 100 == 0 )) ; then
# Display the time
echo -n "$i ";
date +%H:%M:%S.%N
# Count the connections on the NameNode
ssh vagrant@$namenode "file=/tmp/netstat.out ; netstat -a > \$file ; echo
-n 'ESTABLISHED: '; echo -n \`grep -c ESTABLISHED \$file\` ; echo -n '
TIME_WAIT: '; echo -n \`grep -c TIME_WAIT \$file\` ; echo -n ' CLOSE_WAIT: ';
grep -c CLOSE_WAIT \$file"&
fi
# sleep $delay
done
{noformat}
By running one load generator, I got up to 2200 connections, two to 3200.
Also ran the concurrent workload, there's up to 7000 connections. Making
file_size=1 and read_size=1 does not necessarily exacerbate it. I think it has
something to do with my cluster. Three nodes being local VMs. NN/DN/SNN is
equally deployed to 3 nodes.
Let's first try to work on the cache of org.apache.hadoop.ipc.connection to
have a test again.
> Webhdfs client leaks active NameNode connections
>
>
> Key: HDFS-8855
> URL: https://issues.apache.org/jira/browse/HDFS-8855
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
> Environment: HDP 2.2
>Reporter: Bob Hansen
>Assignee: Xiaobing Zhou
>
> The attached script simulates a process opening ~50 files via webhdfs and
> performing random reads. Note that there are at most 50 concurrent reads,
> and all webhdfs sessions are kept open. Each read is ~64k at a random
> position.
> The script periodically (once per second) shells into the NameNode and
> produces a summary of the socket states. For my test cluster with 5 nodes,
> it took ~30 seconds for the NameNode to have ~25000 active connections and
> fails.
> It appears that each request to the webhdfs client is opening a new
> connection to the NameNode and keeping it open after the request is complete.
> If the process continues to run, eventually (~30-60 seconds), all of the
> open connections are closed and the NameNode recovers.
> This smells like SoftReference reaping. Are we using SoftReferences in the
> webhdfs client to cache NameNode connections but never re-using them?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8861) Remove unnecessary log from method FSNamesystem.getCorruptFiles
[
https://issues.apache.org/jira/browse/HDFS-8861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14694121#comment-14694121
]
Tsz Wo Nicholas Sze commented on HDFS-8861:
---
{code}
- LOG.warn("Get corrupt file blocks returned error: " + e.getMessage());
{code}
This line won't print stack trace.
> Remove unnecessary log from method FSNamesystem.getCorruptFiles
> ---
>
> Key: HDFS-8861
> URL: https://issues.apache.org/jira/browse/HDFS-8861
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: HDFS
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Attachments: HDFS-8861.1.patch
>
>
> The log in FSNamesystem.getCorruptFiles will print out whole stack trace. In
> those cases where SuperuserPrivilege check and Operation check are not
> satisfied in frequent calls of listCorruptFileBlocks, the stack traces will
> make logs quite verbose, hard to understand and analyze it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8861) Remove unnecessary log from method FSNamesystem.getCorruptFiles
[ https://issues.apache.org/jira/browse/HDFS-8861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo Nicholas Sze updated HDFS-8861: -- Priority: Minor (was: Major) Component/s: (was: HDFS) namenode > Remove unnecessary log from method FSNamesystem.getCorruptFiles > --- > > Key: HDFS-8861 > URL: https://issues.apache.org/jira/browse/HDFS-8861 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Xiaobing Zhou >Assignee: Xiaobing Zhou >Priority: Minor > Attachments: HDFS-8861.1.patch > > > The log in FSNamesystem.getCorruptFiles will print out whole stack trace. In > those cases where SuperuserPrivilege check and Operation check are not > satisfied in frequent calls of listCorruptFileBlocks, the stack traces will > make logs quite verbose, hard to understand and analyze it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8861) Remove unnecessary log from method FSNamesystem.getCorruptFiles
[ https://issues.apache.org/jira/browse/HDFS-8861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaobing Zhou updated HDFS-8861: Description: The log in FSNamesystem.getCorruptFiles will print out whole stack trace, which makes logs quite verbose, hard to understood and analyzed, especially in those cases where SuperuserPrivilege check and Operation check are not satisfied in frequent calls of listCorruptFileBlocks. (was: The log in FSNamesystem.getCorruptFiles will print out whole stack trace. In those cases where SuperuserPrivilege check and Operation check are not satisfied in frequent calls of listCorruptFileBlocks, the stack traces will make logs quite verbose, hard to understand and analyze it.) > Remove unnecessary log from method FSNamesystem.getCorruptFiles > --- > > Key: HDFS-8861 > URL: https://issues.apache.org/jira/browse/HDFS-8861 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Xiaobing Zhou >Assignee: Xiaobing Zhou >Priority: Minor > Attachments: HDFS-8861.1.patch > > > The log in FSNamesystem.getCorruptFiles will print out whole stack trace, > which makes logs quite verbose, hard to understood and analyzed, especially > in those cases where SuperuserPrivilege check and Operation check are not > satisfied in frequent calls of listCorruptFileBlocks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8861) Remove unnecessary log from method FSNamesystem.getCorruptFiles
[ https://issues.apache.org/jira/browse/HDFS-8861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14694134#comment-14694134 ] Tsz Wo Nicholas Sze commented on HDFS-8861: --- > The log in FSNamesystem.getCorruptFiles will print out whole stack trace, > which makes logs quite verbose Could you give an example of the stack trace? > Remove unnecessary log from method FSNamesystem.getCorruptFiles > --- > > Key: HDFS-8861 > URL: https://issues.apache.org/jira/browse/HDFS-8861 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Xiaobing Zhou >Assignee: Xiaobing Zhou >Priority: Minor > Attachments: HDFS-8861.1.patch > > > The log in FSNamesystem.getCorruptFiles will print out whole stack trace, > which makes logs quite verbose, hard to understood and analyzed, especially > in those cases where SuperuserPrivilege check and Operation check are not > satisfied in frequent calls of listCorruptFileBlocks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8078) HDFS client gets errors trying to to connect to IPv6 DataNode
[ https://issues.apache.org/jira/browse/HDFS-8078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nate Edel updated HDFS-8078: Release Note: was: Resubmitting NetUtils version of patch, with bugfixes. Older version of patch seems to need rebasing, but isn't breaking ZKFC, let's see if these fixes fix that (I can't replicate the break locally.) > HDFS client gets errors trying to to connect to IPv6 DataNode > - > > Key: HDFS-8078 > URL: https://issues.apache.org/jira/browse/HDFS-8078 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Affects Versions: 2.6.0 >Reporter: Nate Edel >Assignee: Nate Edel > Labels: BB2015-05-TBR, ipv6 > Attachments: HDFS-8078.10.patch, HDFS-8078.11.patch, > HDFS-8078.12.patch, HDFS-8078.13.patch, HDFS-8078.14.patch, HDFS-8078.9.patch > > > 1st exception, on put: > 15/03/23 18:43:18 WARN hdfs.DFSClient: DataStreamer Exception > java.lang.IllegalArgumentException: Does not contain a valid host:port > authority: 2401:db00:1010:70ba:face:0:8:0:50010 > at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:212) > at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:164) > at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:153) > at > org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1607) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1408) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1361) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:588) > Appears to actually stem from code in DataNodeID which assumes it's safe to > append together (ipaddr + ":" + port) -- which is OK for IPv4 and not OK for > IPv6. NetUtils.createSocketAddr( ) assembles a Java URI object, which > requires the format proto://[2401:db00:1010:70ba:face:0:8:0]:50010 > Currently using InetAddress.getByName() to validate IPv6 (guava > InetAddresses.forString has been flaky) but could also use our own parsing. > (From logging this, it seems like a low-enough frequency call that the extra > object creation shouldn't be problematic, and for me the slight risk of > passing in bad input that is not actually an IPv4 or IPv6 address and thus > calling an external DNS lookup is outweighed by getting the address > normalized and avoiding rewriting parsing.) > Alternatively, sun.net.util.IPAddressUtil.isIPv6LiteralAddress() > --- > 2nd exception (on datanode) > 15/04/13 13:18:07 ERROR datanode.DataNode: > dev1903.prn1.facebook.com:50010:DataXceiver error processing unknown > operation src: /2401:db00:20:7013:face:0:7:0:54152 dst: > /2401:db00:11:d010:face:0:2f:0:50010 > java.io.EOFException > at java.io.DataInputStream.readShort(DataInputStream.java:315) > at > org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:226) > at java.lang.Thread.run(Thread.java:745) > Which also comes as client error "-get: 2401 is not an IP string literal." > This one has existing parsing logic which needs to shift to the last colon > rather than the first. Should also be a tiny bit faster by using lastIndexOf > rather than split. Could alternatively use the techniques above. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8078) HDFS client gets errors trying to to connect to IPv6 DataNode
[ https://issues.apache.org/jira/browse/HDFS-8078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14694137#comment-14694137 ] Nate Edel commented on HDFS-8078: - Ouch! My apologies. > HDFS client gets errors trying to to connect to IPv6 DataNode > - > > Key: HDFS-8078 > URL: https://issues.apache.org/jira/browse/HDFS-8078 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Affects Versions: 2.6.0 >Reporter: Nate Edel >Assignee: Nate Edel > Labels: BB2015-05-TBR, ipv6 > Attachments: HDFS-8078.10.patch, HDFS-8078.11.patch, > HDFS-8078.12.patch, HDFS-8078.13.patch, HDFS-8078.14.patch, HDFS-8078.9.patch > > > 1st exception, on put: > 15/03/23 18:43:18 WARN hdfs.DFSClient: DataStreamer Exception > java.lang.IllegalArgumentException: Does not contain a valid host:port > authority: 2401:db00:1010:70ba:face:0:8:0:50010 > at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:212) > at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:164) > at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:153) > at > org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1607) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1408) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1361) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:588) > Appears to actually stem from code in DataNodeID which assumes it's safe to > append together (ipaddr + ":" + port) -- which is OK for IPv4 and not OK for > IPv6. NetUtils.createSocketAddr( ) assembles a Java URI object, which > requires the format proto://[2401:db00:1010:70ba:face:0:8:0]:50010 > Currently using InetAddress.getByName() to validate IPv6 (guava > InetAddresses.forString has been flaky) but could also use our own parsing. > (From logging this, it seems like a low-enough frequency call that the extra > object creation shouldn't be problematic, and for me the slight risk of > passing in bad input that is not actually an IPv4 or IPv6 address and thus > calling an external DNS lookup is outweighed by getting the address > normalized and avoiding rewriting parsing.) > Alternatively, sun.net.util.IPAddressUtil.isIPv6LiteralAddress() > --- > 2nd exception (on datanode) > 15/04/13 13:18:07 ERROR datanode.DataNode: > dev1903.prn1.facebook.com:50010:DataXceiver error processing unknown > operation src: /2401:db00:20:7013:face:0:7:0:54152 dst: > /2401:db00:11:d010:face:0:2f:0:50010 > java.io.EOFException > at java.io.DataInputStream.readShort(DataInputStream.java:315) > at > org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:226) > at java.lang.Thread.run(Thread.java:745) > Which also comes as client error "-get: 2401 is not an IP string literal." > This one has existing parsing logic which needs to shift to the last colon > rather than the first. Should also be a tiny bit faster by using lastIndexOf > rather than split. Could alternatively use the techniques above. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8861) Remove unnecessary log from method FSNamesystem.getCorruptFiles
[ https://issues.apache.org/jira/browse/HDFS-8861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaobing Zhou updated HDFS-8861: Description: The log in FSNamesystem.getCorruptFiles will print out too many messages mixed with other log entries, which makes whole log quite verbose, hard to understood and analyzed, especially in those cases where SuperuserPrivilege check and Operation check are not satisfied in frequent calls of listCorruptFileBlocks. (was: The log in FSNamesystem.getCorruptFiles will print out whole stack trace, which makes logs quite verbose, hard to understood and analyzed, especially in those cases where SuperuserPrivilege check and Operation check are not satisfied in frequent calls of listCorruptFileBlocks.) > Remove unnecessary log from method FSNamesystem.getCorruptFiles > --- > > Key: HDFS-8861 > URL: https://issues.apache.org/jira/browse/HDFS-8861 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Xiaobing Zhou >Assignee: Xiaobing Zhou >Priority: Minor > Attachments: HDFS-8861.1.patch > > > The log in FSNamesystem.getCorruptFiles will print out too many messages > mixed with other log entries, which makes whole log quite verbose, hard to > understood and analyzed, especially in those cases where SuperuserPrivilege > check and Operation check are not satisfied in frequent calls of > listCorruptFileBlocks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8861) Remove unnecessary log from method FSNamesystem.getCorruptFiles
[ https://issues.apache.org/jira/browse/HDFS-8861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14694192#comment-14694192 ] Xiaobing Zhou commented on HDFS-8861: - Removed inaccurate description. > Remove unnecessary log from method FSNamesystem.getCorruptFiles > --- > > Key: HDFS-8861 > URL: https://issues.apache.org/jira/browse/HDFS-8861 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Xiaobing Zhou >Assignee: Xiaobing Zhou >Priority: Minor > Attachments: HDFS-8861.1.patch > > > The log in FSNamesystem.getCorruptFiles will print out too many messages > mixed with other log entries, which makes whole log quite verbose, hard to > understood and analyzed, especially in those cases where SuperuserPrivilege > check and Operation check are not satisfied in frequent calls of > listCorruptFileBlocks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8861) Remove unnecessary log from method FSNamesystem.getCorruptFiles
[ https://issues.apache.org/jira/browse/HDFS-8861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14694220#comment-14694220 ] Jitendra Nath Pandey commented on HDFS-8861: It seems HDFS-8522 addresses this issue. > Remove unnecessary log from method FSNamesystem.getCorruptFiles > --- > > Key: HDFS-8861 > URL: https://issues.apache.org/jira/browse/HDFS-8861 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Xiaobing Zhou >Assignee: Xiaobing Zhou >Priority: Minor > Attachments: HDFS-8861.1.patch > > > The log in FSNamesystem.getCorruptFiles will print out too many messages > mixed with other log entries, which makes whole log quite verbose, hard to > understood and analyzed, especially in those cases where SuperuserPrivilege > check and Operation check are not satisfied in frequent calls of > listCorruptFileBlocks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8883) NameNode Metrics : Add FSNameSystem lock Queue Length
[
https://issues.apache.org/jira/browse/HDFS-8883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14694229#comment-14694229
]
Hadoop QA commented on HDFS-8883:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 19m 50s | Findbugs (version ) appears to
be broken on trunk. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 2 new or modified test files. |
| {color:green}+1{color} | javac | 7m 37s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 41s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | site | 2m 57s | Site still builds. |
| {color:green}+1{color} | checkstyle | 1m 38s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 31s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 4m 21s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:red}-1{color} | common tests | 22m 21s | Tests failed in
hadoop-common. |
| {color:red}-1{color} | hdfs tests | 176m 4s | Tests failed in hadoop-hdfs. |
| | | 246m 58s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.ha.TestZKFailoverController |
| | hadoop.net.TestNetUtils |
| | hadoop.hdfs.TestRollingUpgrade |
| Timed out tests | org.apache.hadoop.cli.TestHDFSCLI |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12749711/HDFS-8883.001.patch |
| Optional Tests | site javadoc javac unit findbugs checkstyle |
| git revision | trunk / 1c12adb |
| hadoop-common test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/11982/artifact/patchprocess/testrun_hadoop-common.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/11982/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/11982/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/11982/console |
This message was automatically generated.
> NameNode Metrics : Add FSNameSystem lock Queue Length
> -
>
> Key: HDFS-8883
> URL: https://issues.apache.org/jira/browse/HDFS-8883
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: HDFS
>Affects Versions: 2.7.1
>Reporter: Anu Engineer
>Assignee: Anu Engineer
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HDFS-8883.001.patch
>
>
> FSNameSystemLock can have contention when NameNode is under load. This patch
> adds LockQueueLength -- the number of threads waiting on FSNameSystemLock --
> as a metric in NameNode.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-8890) Allow admin to specify which blockpools the balancer should run on
Chris Trezzo created HDFS-8890: -- Summary: Allow admin to specify which blockpools the balancer should run on Key: HDFS-8890 URL: https://issues.apache.org/jira/browse/HDFS-8890 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Chris Trezzo Assignee: Chris Trezzo Currently the balancer runs on all blockpools. Allow an admin to run the balancer on a set of blockpools. This will enable the balancer to skip blockpools that should not be balanced. For example, a tmp blockpool that has a large amount of churn. An example of the command line interface would be an additional flag that specifies the blockpools by id: -blockpools BP-6299761-10.55.116.188-1415904647555,BP-47348528-10.51.120.139-1415904199257 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (HDFS-8890) Allow admin to specify which blockpools the balancer should run on
[ https://issues.apache.org/jira/browse/HDFS-8890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDFS-8890 started by Chris Trezzo. -- > Allow admin to specify which blockpools the balancer should run on > -- > > Key: HDFS-8890 > URL: https://issues.apache.org/jira/browse/HDFS-8890 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: balancer & mover >Reporter: Chris Trezzo >Assignee: Chris Trezzo > > Currently the balancer runs on all blockpools. Allow an admin to run the > balancer on a set of blockpools. This will enable the balancer to skip > blockpools that should not be balanced. For example, a tmp blockpool that has > a large amount of churn. > An example of the command line interface would be an additional flag that > specifies the blockpools by id: > -blockpools > BP-6299761-10.55.116.188-1415904647555,BP-47348528-10.51.120.139-1415904199257 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8861) Remove unnecessary log from method FSNamesystem.getCorruptFiles
[ https://issues.apache.org/jira/browse/HDFS-8861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14694247#comment-14694247 ] Xiaobing Zhou commented on HDFS-8861: - No, HDFS-8522 didn't handle getCorruptFiles. Thanks [~szetszwo] and [~jnp] for review. > Remove unnecessary log from method FSNamesystem.getCorruptFiles > --- > > Key: HDFS-8861 > URL: https://issues.apache.org/jira/browse/HDFS-8861 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Xiaobing Zhou >Assignee: Xiaobing Zhou >Priority: Minor > Attachments: HDFS-8861.1.patch > > > The log in FSNamesystem.getCorruptFiles will print out too many messages > mixed with other log entries, which makes whole log quite verbose, hard to > understood and analyzed, especially in those cases where SuperuserPrivilege > check and Operation check are not satisfied in frequent calls of > listCorruptFileBlocks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8883) NameNode Metrics : Add FSNameSystem lock Queue Length
[ https://issues.apache.org/jira/browse/HDFS-8883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14694273#comment-14694273 ] Anu Engineer commented on HDFS-8883: None of the test failures seem to be related to this patch. Re-running them on my local machine does succeed. > NameNode Metrics : Add FSNameSystem lock Queue Length > - > > Key: HDFS-8883 > URL: https://issues.apache.org/jira/browse/HDFS-8883 > Project: Hadoop HDFS > Issue Type: Improvement > Components: HDFS >Affects Versions: 2.7.1 >Reporter: Anu Engineer >Assignee: Anu Engineer >Priority: Minor > Fix For: 2.8.0 > > Attachments: HDFS-8883.001.patch > > > FSNameSystemLock can have contention when NameNode is under load. This patch > adds LockQueueLength -- the number of threads waiting on FSNameSystemLock -- > as a metric in NameNode. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8823) Move replication factor into individual blocks
[
https://issues.apache.org/jira/browse/HDFS-8823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haohui Mai updated HDFS-8823:
-
Attachment: HDFS-8823.003.patch
> Move replication factor into individual blocks
> --
>
> Key: HDFS-8823
> URL: https://issues.apache.org/jira/browse/HDFS-8823
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Attachments: HDFS-8823.000.patch, HDFS-8823.001.patch,
> HDFS-8823.002.patch, HDFS-8823.003.patch
>
>
> This jira proposes to record the replication factor in the {{BlockInfo}}
> class. The changes have two advantages:
> * Decoupling the namespace and the block management layer. It is a
> prerequisite step to move block management off the heap or to a separate
> process.
> * Increased flexibility on replicating blocks. Currently the replication
> factors of all blocks have to be the same. The replication factors of these
> blocks are equal to the highest replication factor across all snapshots. The
> changes will allow blocks in a file to have different replication factor,
> potentially saving some space.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8052) Move WebHdfsFileSystem into hadoop-hdfs-client
[ https://issues.apache.org/jira/browse/HDFS-8052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14694310#comment-14694310 ] Victor Malov commented on HDFS-8052: - Am I correct that this WebHdfsFilesystem implements Java Filesystem class and it will be possible to use it in standard Java way with Files/Filesystem/Path classes? Will it be public? I couldn't find any mention of this in Hadoop 3 documentation. And what about making it as separate standalone project? For example if someone decides to write filemanager like Midnight Commander for HDFS and it no needs whole bunch of Hadoop jars. Just one dependency on this. > Move WebHdfsFileSystem into hadoop-hdfs-client > -- > > Key: HDFS-8052 > URL: https://issues.apache.org/jira/browse/HDFS-8052 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: build >Reporter: Haohui Mai >Assignee: Haohui Mai > Fix For: 2.8.0 > > Attachments: HDFS-8052.000.patch, HDFS-8052.001.patch, > HDFS-8052.002.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8879) Quota by storage type usage incorrectly initialized upon namenode restart
[
https://issues.apache.org/jira/browse/HDFS-8879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoyu Yao updated HDFS-8879:
-
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 2.8.0
Status: Resolved (was: Patch Available)
Thanks all for the review. I've checked in the fix to trunk and branch-2.
> Quota by storage type usage incorrectly initialized upon namenode restart
> -
>
> Key: HDFS-8879
> URL: https://issues.apache.org/jira/browse/HDFS-8879
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.0
>Reporter: Kihwal Lee
>Assignee: Xiaoyu Yao
> Fix For: 2.8.0
>
> Attachments: HDFS-8879.01.patch
>
>
> This was found by [~kihwal] as part of HDFS-8865 work in this
> [comment|https://issues.apache.org/jira/browse/HDFS-8865?focusedCommentId=14660904&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14660904].
> The unit test
> testQuotaByStorageTypePersistenceInFsImage/testQuotaByStorageTypePersistenceInFsEdit
> failed to detect this because they were using an obsolete
> FsDirectory instance. Once added the highlighted line below, the issue can be
> reproed.
> {code}
> >fsdir = cluster.getNamesystem().getFSDirectory();
> INode testDirNodeAfterNNRestart = fsdir.getINode4Write(testDir.toString());
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8823) Move replication factor into individual blocks
[
https://issues.apache.org/jira/browse/HDFS-8823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haohui Mai updated HDFS-8823:
-
Attachment: HDFS-8823.004.patch
> Move replication factor into individual blocks
> --
>
> Key: HDFS-8823
> URL: https://issues.apache.org/jira/browse/HDFS-8823
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Attachments: HDFS-8823.000.patch, HDFS-8823.001.patch,
> HDFS-8823.002.patch, HDFS-8823.003.patch, HDFS-8823.004.patch
>
>
> This jira proposes to record the replication factor in the {{BlockInfo}}
> class. The changes have two advantages:
> * Decoupling the namespace and the block management layer. It is a
> prerequisite step to move block management off the heap or to a separate
> process.
> * Increased flexibility on replicating blocks. Currently the replication
> factors of all blocks have to be the same. The replication factors of these
> blocks are equal to the highest replication factor across all snapshots. The
> changes will allow blocks in a file to have different replication factor,
> potentially saving some space.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8052) Move WebHdfsFileSystem into hadoop-hdfs-client
[ https://issues.apache.org/jira/browse/HDFS-8052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14694314#comment-14694314 ] Victor Malov commented on HDFS-8052: - In this way it could just install one simple dependency WebHdfsFilesystem, without even installing Hadoop and use it to get HDFS-like filesystems. > Move WebHdfsFileSystem into hadoop-hdfs-client > -- > > Key: HDFS-8052 > URL: https://issues.apache.org/jira/browse/HDFS-8052 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: build >Reporter: Haohui Mai >Assignee: Haohui Mai > Fix For: 2.8.0 > > Attachments: HDFS-8052.000.patch, HDFS-8052.001.patch, > HDFS-8052.002.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8879) Quota by storage type usage incorrectly initialized upon namenode restart
[
https://issues.apache.org/jira/browse/HDFS-8879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14694339#comment-14694339
]
Hudson commented on HDFS-8879:
--
FAILURE: Integrated in Hadoop-trunk-Commit #8289 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/8289/])
HDFS-8879. Quota by storage type usage incorrectly initialized upon namenode
restart. Contributed by Xiaoyu Yao. (xyao: rev
3e715a4f4c46bcd8b3054cb0566e526c46bd5d66)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestQuotaByStorageType.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
> Quota by storage type usage incorrectly initialized upon namenode restart
> -
>
> Key: HDFS-8879
> URL: https://issues.apache.org/jira/browse/HDFS-8879
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.0
>Reporter: Kihwal Lee
>Assignee: Xiaoyu Yao
> Fix For: 2.8.0
>
> Attachments: HDFS-8879.01.patch
>
>
> This was found by [~kihwal] as part of HDFS-8865 work in this
> [comment|https://issues.apache.org/jira/browse/HDFS-8865?focusedCommentId=14660904&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14660904].
> The unit test
> testQuotaByStorageTypePersistenceInFsImage/testQuotaByStorageTypePersistenceInFsEdit
> failed to detect this because they were using an obsolete
> FsDirectory instance. Once added the highlighted line below, the issue can be
> reproed.
> {code}
> >fsdir = cluster.getNamesystem().getFSDirectory();
> INode testDirNodeAfterNNRestart = fsdir.getINode4Write(testDir.toString());
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8078) HDFS client gets errors trying to to connect to IPv6 DataNode
[ https://issues.apache.org/jira/browse/HDFS-8078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nate Edel updated HDFS-8078: Status: Open (was: Patch Available) > HDFS client gets errors trying to to connect to IPv6 DataNode > - > > Key: HDFS-8078 > URL: https://issues.apache.org/jira/browse/HDFS-8078 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Affects Versions: 2.6.0 >Reporter: Nate Edel >Assignee: Nate Edel > Labels: BB2015-05-TBR, ipv6 > Attachments: HDFS-8078.10.patch, HDFS-8078.11.patch, > HDFS-8078.12.patch, HDFS-8078.13.patch, HDFS-8078.14.patch, HDFS-8078.9.patch > > > 1st exception, on put: > 15/03/23 18:43:18 WARN hdfs.DFSClient: DataStreamer Exception > java.lang.IllegalArgumentException: Does not contain a valid host:port > authority: 2401:db00:1010:70ba:face:0:8:0:50010 > at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:212) > at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:164) > at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:153) > at > org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1607) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1408) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1361) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:588) > Appears to actually stem from code in DataNodeID which assumes it's safe to > append together (ipaddr + ":" + port) -- which is OK for IPv4 and not OK for > IPv6. NetUtils.createSocketAddr( ) assembles a Java URI object, which > requires the format proto://[2401:db00:1010:70ba:face:0:8:0]:50010 > Currently using InetAddress.getByName() to validate IPv6 (guava > InetAddresses.forString has been flaky) but could also use our own parsing. > (From logging this, it seems like a low-enough frequency call that the extra > object creation shouldn't be problematic, and for me the slight risk of > passing in bad input that is not actually an IPv4 or IPv6 address and thus > calling an external DNS lookup is outweighed by getting the address > normalized and avoiding rewriting parsing.) > Alternatively, sun.net.util.IPAddressUtil.isIPv6LiteralAddress() > --- > 2nd exception (on datanode) > 15/04/13 13:18:07 ERROR datanode.DataNode: > dev1903.prn1.facebook.com:50010:DataXceiver error processing unknown > operation src: /2401:db00:20:7013:face:0:7:0:54152 dst: > /2401:db00:11:d010:face:0:2f:0:50010 > java.io.EOFException > at java.io.DataInputStream.readShort(DataInputStream.java:315) > at > org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:226) > at java.lang.Thread.run(Thread.java:745) > Which also comes as client error "-get: 2401 is not an IP string literal." > This one has existing parsing logic which needs to shift to the last colon > rather than the first. Should also be a tiny bit faster by using lastIndexOf > rather than split. Could alternatively use the techniques above. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8078) HDFS client gets errors trying to to connect to IPv6 DataNode
[ https://issues.apache.org/jira/browse/HDFS-8078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nate Edel updated HDFS-8078: Status: Patch Available (was: Open) > HDFS client gets errors trying to to connect to IPv6 DataNode > - > > Key: HDFS-8078 > URL: https://issues.apache.org/jira/browse/HDFS-8078 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Affects Versions: 2.6.0 >Reporter: Nate Edel >Assignee: Nate Edel > Labels: BB2015-05-TBR, ipv6 > Attachments: HDFS-8078.10.patch, HDFS-8078.11.patch, > HDFS-8078.12.patch, HDFS-8078.13.patch, HDFS-8078.14.patch, > HDFS-8078.9.patch, dummy.patch > > > 1st exception, on put: > 15/03/23 18:43:18 WARN hdfs.DFSClient: DataStreamer Exception > java.lang.IllegalArgumentException: Does not contain a valid host:port > authority: 2401:db00:1010:70ba:face:0:8:0:50010 > at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:212) > at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:164) > at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:153) > at > org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1607) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1408) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1361) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:588) > Appears to actually stem from code in DataNodeID which assumes it's safe to > append together (ipaddr + ":" + port) -- which is OK for IPv4 and not OK for > IPv6. NetUtils.createSocketAddr( ) assembles a Java URI object, which > requires the format proto://[2401:db00:1010:70ba:face:0:8:0]:50010 > Currently using InetAddress.getByName() to validate IPv6 (guava > InetAddresses.forString has been flaky) but could also use our own parsing. > (From logging this, it seems like a low-enough frequency call that the extra > object creation shouldn't be problematic, and for me the slight risk of > passing in bad input that is not actually an IPv4 or IPv6 address and thus > calling an external DNS lookup is outweighed by getting the address > normalized and avoiding rewriting parsing.) > Alternatively, sun.net.util.IPAddressUtil.isIPv6LiteralAddress() > --- > 2nd exception (on datanode) > 15/04/13 13:18:07 ERROR datanode.DataNode: > dev1903.prn1.facebook.com:50010:DataXceiver error processing unknown > operation src: /2401:db00:20:7013:face:0:7:0:54152 dst: > /2401:db00:11:d010:face:0:2f:0:50010 > java.io.EOFException > at java.io.DataInputStream.readShort(DataInputStream.java:315) > at > org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:226) > at java.lang.Thread.run(Thread.java:745) > Which also comes as client error "-get: 2401 is not an IP string literal." > This one has existing parsing logic which needs to shift to the last colon > rather than the first. Should also be a tiny bit faster by using lastIndexOf > rather than split. Could alternatively use the techniques above. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8078) HDFS client gets errors trying to to connect to IPv6 DataNode
[ https://issues.apache.org/jira/browse/HDFS-8078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nate Edel updated HDFS-8078: Attachment: dummy.patch Dummy patch to NetUtils.java to see if we break the same tests with no functional changes... > HDFS client gets errors trying to to connect to IPv6 DataNode > - > > Key: HDFS-8078 > URL: https://issues.apache.org/jira/browse/HDFS-8078 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Affects Versions: 2.6.0 >Reporter: Nate Edel >Assignee: Nate Edel > Labels: BB2015-05-TBR, ipv6 > Attachments: HDFS-8078.10.patch, HDFS-8078.11.patch, > HDFS-8078.12.patch, HDFS-8078.13.patch, HDFS-8078.14.patch, > HDFS-8078.9.patch, dummy.patch > > > 1st exception, on put: > 15/03/23 18:43:18 WARN hdfs.DFSClient: DataStreamer Exception > java.lang.IllegalArgumentException: Does not contain a valid host:port > authority: 2401:db00:1010:70ba:face:0:8:0:50010 > at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:212) > at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:164) > at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:153) > at > org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1607) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1408) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1361) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:588) > Appears to actually stem from code in DataNodeID which assumes it's safe to > append together (ipaddr + ":" + port) -- which is OK for IPv4 and not OK for > IPv6. NetUtils.createSocketAddr( ) assembles a Java URI object, which > requires the format proto://[2401:db00:1010:70ba:face:0:8:0]:50010 > Currently using InetAddress.getByName() to validate IPv6 (guava > InetAddresses.forString has been flaky) but could also use our own parsing. > (From logging this, it seems like a low-enough frequency call that the extra > object creation shouldn't be problematic, and for me the slight risk of > passing in bad input that is not actually an IPv4 or IPv6 address and thus > calling an external DNS lookup is outweighed by getting the address > normalized and avoiding rewriting parsing.) > Alternatively, sun.net.util.IPAddressUtil.isIPv6LiteralAddress() > --- > 2nd exception (on datanode) > 15/04/13 13:18:07 ERROR datanode.DataNode: > dev1903.prn1.facebook.com:50010:DataXceiver error processing unknown > operation src: /2401:db00:20:7013:face:0:7:0:54152 dst: > /2401:db00:11:d010:face:0:2f:0:50010 > java.io.EOFException > at java.io.DataInputStream.readShort(DataInputStream.java:315) > at > org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:226) > at java.lang.Thread.run(Thread.java:745) > Which also comes as client error "-get: 2401 is not an IP string literal." > This one has existing parsing logic which needs to shift to the last colon > rather than the first. Should also be a tiny bit faster by using lastIndexOf > rather than split. Could alternatively use the techniques above. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8155) Support OAuth2 in WebHDFS
[
https://issues.apache.org/jira/browse/HDFS-8155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14694416#comment-14694416
]
Chris Douglas commented on HDFS-8155:
-
Minor points; someone should look at the OAuth changes specifically.
* Instead of adding its own {{Time}} classes under utils, it may make sense to
move
[Clock|https://git1-us-west.apache.org/repos/asf?p=hadoop.git;a=blob;f=hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/Clock.java;h=fe55993d5db520b542e21bfa7e2dfb82011a60bc;hb=refs/heads/trunk]
from YARN to Common. Or use the
[Timer|https://git1-us-west.apache.org/repos/asf?p=hadoop.git;a=blob;f=hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/Timer.java;h=e1e21a7220013016c443923331483489641d1306;hb=refs/heads/trunk]
class in Common, if that's sufficient.
* If the timing classes could use existing code, the Utils class could be
package-private in {{o.a.h.hdfs.web.oauth2}}
* Please add scope/visibility annotations to classes
* The protected, mutable fields in the {{AccessTokenProvider}} classes can be
private. {{nextRefreshMSSinceEpoch}} in {{Timer}}, also (constructor should
explicitly init to 0 instead of self-ref)
* Instead of requiring {{initialize}}, would it make sense for
{{AccessTokenProvider}} implementations to implement {{Configurable}} as
appropriate?
* What is the expectation for multithreaded access for these classes?
* If {{AccessTokenProvider}} were an abstract class, could the impls share more
of the code in refresh? Superficially, they look very similar...
* Should refresh throw {{IOException}} instead of {{IllegalArgumentException}},
since {{AccessTokenProvider::getAccessToken}} supports it?
> Support OAuth2 in WebHDFS
> -
>
> Key: HDFS-8155
> URL: https://issues.apache.org/jira/browse/HDFS-8155
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: webhdfs
>Reporter: Jakob Homan
>Assignee: Jakob Homan
> Attachments: HDFS-8155-1.patch
>
>
> WebHDFS should be able to accept OAuth2 credentials.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8791) block ID-based DN storage layout can be very slow for datanode on ext4
[ https://issues.apache.org/jira/browse/HDFS-8791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14694504#comment-14694504 ] Joep Rottinghuis commented on HDFS-8791: Seems related to (or perhaps dup of) HADOOP-10434. > block ID-based DN storage layout can be very slow for datanode on ext4 > -- > > Key: HDFS-8791 > URL: https://issues.apache.org/jira/browse/HDFS-8791 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.6.0 >Reporter: Nathan Roberts >Priority: Critical > > We are seeing cases where the new directory layout causes the datanode to > basically cause the disks to seek for 10s of minutes. This can be when the > datanode is running du, and it can also be when it is performing a > checkDirs(). Both of these operations currently scan all directories in the > block pool and that's very expensive in the new layout. > The new layout creates 256 subdirs, each with 256 subdirs. Essentially 64K > leaf directories where block files are placed. > So, what we have on disk is: > - 256 inodes for the first level directories > - 256 directory blocks for the first level directories > - 256*256 inodes for the second level directories > - 256*256 directory blocks for the second level directories > - Then the inodes and blocks to store the the HDFS blocks themselves. > The main problem is the 256*256 directory blocks. > inodes and dentries will be cached by linux and one can configure how likely > the system is to prune those entries (vfs_cache_pressure). However, ext4 > relies on the buffer cache to cache the directory blocks and I'm not aware of > any way to tell linux to favor buffer cache pages (even if it did I'm not > sure I would want it to in general). > Also, ext4 tries hard to spread directories evenly across the entire volume, > this basically means the 64K directory blocks are probably randomly spread > across the entire disk. A du type scan will look at directories one at a > time, so the ioscheduler can't optimize the corresponding seeks, meaning the > seeks will be random and far. > In a system I was using to diagnose this, I had 60K blocks. A DU when things > are hot is less than 1 second. When things are cold, about 20 minutes. > How do things get cold? > - A large set of tasks run on the node. This pushes almost all of the buffer > cache out, causing the next DU to hit this situation. We are seeing cases > where a large job can cause a seek storm across the entire cluster. > Why didn't the previous layout see this? > - It might have but it wasn't nearly as pronounced. The previous layout would > be a few hundred directory blocks. Even when completely cold, these would > only take a few a hundred seeks which would mean single digit seconds. > - With only a few hundred directories, the odds of the directory blocks > getting modified is quite high, this keeps those blocks hot and much less > likely to be evicted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8078) HDFS client gets errors trying to to connect to IPv6 DataNode
[
https://issues.apache.org/jira/browse/HDFS-8078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14694522#comment-14694522
]
Hadoop QA commented on HDFS-8078:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 17m 4s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | javac | 7m 55s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 50s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 24s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 1m 14s | There were no new checkstyle
issues. |
| {color:red}-1{color} | whitespace | 0m 0s | The patch has 1 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 22s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 58s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:red}-1{color} | common tests | 22m 46s | Tests failed in
hadoop-common. |
| | | 63m 11s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.ha.TestZKFailoverController |
| | hadoop.net.TestNetUtils |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12750193/dummy.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 6cc8e38 |
| whitespace |
https://builds.apache.org/job/PreCommit-HDFS-Build/11984/artifact/patchprocess/whitespace.txt
|
| hadoop-common test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/11984/artifact/patchprocess/testrun_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/11984/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/11984/console |
This message was automatically generated.
> HDFS client gets errors trying to to connect to IPv6 DataNode
> -
>
> Key: HDFS-8078
> URL: https://issues.apache.org/jira/browse/HDFS-8078
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Affects Versions: 2.6.0
>Reporter: Nate Edel
>Assignee: Nate Edel
> Labels: BB2015-05-TBR, ipv6
> Attachments: HDFS-8078.10.patch, HDFS-8078.11.patch,
> HDFS-8078.12.patch, HDFS-8078.13.patch, HDFS-8078.14.patch,
> HDFS-8078.9.patch, dummy.patch
>
>
> 1st exception, on put:
> 15/03/23 18:43:18 WARN hdfs.DFSClient: DataStreamer Exception
> java.lang.IllegalArgumentException: Does not contain a valid host:port
> authority: 2401:db00:1010:70ba:face:0:8:0:50010
> at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:212)
> at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:164)
> at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:153)
> at
> org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1607)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1408)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1361)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:588)
> Appears to actually stem from code in DataNodeID which assumes it's safe to
> append together (ipaddr + ":" + port) -- which is OK for IPv4 and not OK for
> IPv6. NetUtils.createSocketAddr( ) assembles a Java URI object, which
> requires the format proto://[2401:db00:1010:70ba:face:0:8:0]:50010
> Currently using InetAddress.getByName() to validate IPv6 (guava
> InetAddresses.forString has been flaky) but could also use our own parsing.
> (From logging this, it seems like a low-enough frequency call that the extra
> object creation shouldn't be problematic, and for me the slight risk of
> passing in bad input that is not actually an IPv4 or IPv6 address and thus
> calling an external DNS lookup is outweighed by getting the address
> normalized and avoiding rewriti
[jira] [Commented] (HDFS-8854) Erasure coding: add ECPolicy to replace schema+cellSize in hadoop-hdfs
[ https://issues.apache.org/jira/browse/HDFS-8854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14694545#comment-14694545 ] Allen Wittenauer commented on HDFS-8854: That's because the JIRA is also a branch name so since the patch doesn't contain a branch, it tries the JIRA. So name it blah.trunk.##.patch . > Erasure coding: add ECPolicy to replace schema+cellSize in hadoop-hdfs > -- > > Key: HDFS-8854 > URL: https://issues.apache.org/jira/browse/HDFS-8854 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7285 >Reporter: Walter Su >Assignee: Walter Su > Attachments: HDFS-8854-Consolidated-20150806.02.txt, > HDFS-8854-HDFS-7285-merge.03.patch, HDFS-8854-HDFS-7285-merge.03.txt, > HDFS-8854-HDFS-7285.00.patch, HDFS-8854-HDFS-7285.01.patch, > HDFS-8854-HDFS-7285.02.patch, HDFS-8854-HDFS-7285.03.patch, HDFS-8854.00.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8823) Move replication factor into individual blocks
[
https://issues.apache.org/jira/browse/HDFS-8823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14694556#comment-14694556
]
Hadoop QA commented on HDFS-8823:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 15m 41s | Findbugs (version ) appears to
be broken on trunk. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 7 new or modified test files. |
| {color:green}+1{color} | javac | 7m 49s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 45s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 26s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 41s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 7s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 32s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:red}-1{color} | findbugs | 2m 38s | The patch appears to introduce 1
new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 6s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 114m 29s | Tests failed in hadoop-hdfs. |
| | | 156m 53s | |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs |
| Failed unit tests | hadoop.hdfs.TestQuota |
| | hadoop.hdfs.server.namenode.snapshot.TestSnapshotDeletion |
| | hadoop.hdfs.TestDistributedFileSystem |
| | hadoop.hdfs.server.namenode.TestFileContextAcl |
| Timed out tests | org.apache.hadoop.hdfs.server.namenode.TestAddBlockRetry |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12750178/HDFS-8823.004.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / dc2340c |
| Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/11983/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/11983/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/11983/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/11983/console |
This message was automatically generated.
> Move replication factor into individual blocks
> --
>
> Key: HDFS-8823
> URL: https://issues.apache.org/jira/browse/HDFS-8823
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Attachments: HDFS-8823.000.patch, HDFS-8823.001.patch,
> HDFS-8823.002.patch, HDFS-8823.003.patch, HDFS-8823.004.patch
>
>
> This jira proposes to record the replication factor in the {{BlockInfo}}
> class. The changes have two advantages:
> * Decoupling the namespace and the block management layer. It is a
> prerequisite step to move block management off the heap or to a separate
> process.
> * Increased flexibility on replicating blocks. Currently the replication
> factors of all blocks have to be the same. The replication factors of these
> blocks are equal to the highest replication factor across all snapshots. The
> changes will allow blocks in a file to have different replication factor,
> potentially saving some space.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8078) HDFS client gets errors trying to to connect to IPv6 DataNode
[ https://issues.apache.org/jira/browse/HDFS-8078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14694557#comment-14694557 ] Nate Edel commented on HDFS-8078: - OK, the tests above are clearly broken in trunk, since that was a no-op patch. Any suggestions on how to fix these? > HDFS client gets errors trying to to connect to IPv6 DataNode > - > > Key: HDFS-8078 > URL: https://issues.apache.org/jira/browse/HDFS-8078 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Affects Versions: 2.6.0 >Reporter: Nate Edel >Assignee: Nate Edel > Labels: BB2015-05-TBR, ipv6 > Attachments: HDFS-8078.10.patch, HDFS-8078.11.patch, > HDFS-8078.12.patch, HDFS-8078.13.patch, HDFS-8078.14.patch, > HDFS-8078.9.patch, dummy.patch > > > 1st exception, on put: > 15/03/23 18:43:18 WARN hdfs.DFSClient: DataStreamer Exception > java.lang.IllegalArgumentException: Does not contain a valid host:port > authority: 2401:db00:1010:70ba:face:0:8:0:50010 > at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:212) > at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:164) > at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:153) > at > org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1607) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1408) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1361) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:588) > Appears to actually stem from code in DataNodeID which assumes it's safe to > append together (ipaddr + ":" + port) -- which is OK for IPv4 and not OK for > IPv6. NetUtils.createSocketAddr( ) assembles a Java URI object, which > requires the format proto://[2401:db00:1010:70ba:face:0:8:0]:50010 > Currently using InetAddress.getByName() to validate IPv6 (guava > InetAddresses.forString has been flaky) but could also use our own parsing. > (From logging this, it seems like a low-enough frequency call that the extra > object creation shouldn't be problematic, and for me the slight risk of > passing in bad input that is not actually an IPv4 or IPv6 address and thus > calling an external DNS lookup is outweighed by getting the address > normalized and avoiding rewriting parsing.) > Alternatively, sun.net.util.IPAddressUtil.isIPv6LiteralAddress() > --- > 2nd exception (on datanode) > 15/04/13 13:18:07 ERROR datanode.DataNode: > dev1903.prn1.facebook.com:50010:DataXceiver error processing unknown > operation src: /2401:db00:20:7013:face:0:7:0:54152 dst: > /2401:db00:11:d010:face:0:2f:0:50010 > java.io.EOFException > at java.io.DataInputStream.readShort(DataInputStream.java:315) > at > org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:226) > at java.lang.Thread.run(Thread.java:745) > Which also comes as client error "-get: 2401 is not an IP string literal." > This one has existing parsing logic which needs to shift to the last colon > rather than the first. Should also be a tiny bit faster by using lastIndexOf > rather than split. Could alternatively use the techniques above. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7649) Multihoming docs should emphasize using hostnames in configurations
[ https://issues.apache.org/jira/browse/HDFS-7649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-7649: --- Attachment: HDFS-7649.patch > Multihoming docs should emphasize using hostnames in configurations > --- > > Key: HDFS-7649 > URL: https://issues.apache.org/jira/browse/HDFS-7649 > Project: Hadoop HDFS > Issue Type: Improvement > Components: documentation >Reporter: Arpit Agarwal >Assignee: Brahma Reddy Battula > Attachments: HDFS-7649.patch > > > The docs should emphasize that master and slave configurations should > hostnames wherever possible. > Link to current docs: > https://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-hdfs/HdfsMultihoming.html -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7649) Multihoming docs should emphasize using hostnames in configurations
[ https://issues.apache.org/jira/browse/HDFS-7649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-7649: --- Status: Patch Available (was: Open) > Multihoming docs should emphasize using hostnames in configurations > --- > > Key: HDFS-7649 > URL: https://issues.apache.org/jira/browse/HDFS-7649 > Project: Hadoop HDFS > Issue Type: Improvement > Components: documentation >Reporter: Arpit Agarwal >Assignee: Brahma Reddy Battula > Attachments: HDFS-7649.patch > > > The docs should emphasize that master and slave configurations should > hostnames wherever possible. > Link to current docs: > https://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-hdfs/HdfsMultihoming.html -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7649) Multihoming docs should emphasize using hostnames in configurations
[ https://issues.apache.org/jira/browse/HDFS-7649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14694568#comment-14694568 ] Brahma Reddy Battula commented on HDFS-7649: [~arpitagarwal] Based on your input, I uploaded the patch..Kindly review.. thanks!! > Multihoming docs should emphasize using hostnames in configurations > --- > > Key: HDFS-7649 > URL: https://issues.apache.org/jira/browse/HDFS-7649 > Project: Hadoop HDFS > Issue Type: Improvement > Components: documentation >Reporter: Arpit Agarwal >Assignee: Brahma Reddy Battula > Attachments: HDFS-7649.patch > > > The docs should emphasize that master and slave configurations should > hostnames wherever possible. > Link to current docs: > https://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-hdfs/HdfsMultihoming.html -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8622) Implement GETCONTENTSUMMARY operation for WebImageViewer
[
https://issues.apache.org/jira/browse/HDFS-8622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14694575#comment-14694575
]
Akira AJISAKA commented on HDFS-8622:
-
+1, the test failure looks unrelated to the patch. I ran the test successfully
on my environment.
> Implement GETCONTENTSUMMARY operation for WebImageViewer
>
>
> Key: HDFS-8622
> URL: https://issues.apache.org/jira/browse/HDFS-8622
> Project: Hadoop HDFS
> Issue Type: New Feature
>Reporter: Jagadesh Kiran N
>Assignee: Jagadesh Kiran N
> Attachments: HDFS-8622-00.patch, HDFS-8622-01.patch,
> HDFS-8622-02.patch, HDFS-8622-03.patch, HDFS-8622-04.patch,
> HDFS-8622-05.patch, HDFS-8622-06.patch, HDFS-8622-07.patch,
> HDFS-8622-08.patch, HDFS-8622-09.patch, HDFS-8622-10.patch
>
>
> it would be better for administrators if {code} GETCONTENTSUMMARY {code} are
> supported.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7649) Multihoming docs should emphasize using hostnames in configurations
[
https://issues.apache.org/jira/browse/HDFS-7649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14694637#comment-14694637
]
Hadoop QA commented on HDFS-7649:
-
\\
\\
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 3m 0s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | release audit | 0m 20s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | site | 3m 0s | Site still builds. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| | | 6m 23s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12750221/HDFS-7649.patch |
| Optional Tests | site |
| git revision | trunk / 6cc8e38 |
| Java | 1.7.0_55 |
| uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/11985/console |
This message was automatically generated.
> Multihoming docs should emphasize using hostnames in configurations
> ---
>
> Key: HDFS-7649
> URL: https://issues.apache.org/jira/browse/HDFS-7649
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: documentation
>Reporter: Arpit Agarwal
>Assignee: Brahma Reddy Battula
> Attachments: HDFS-7649.patch
>
>
> The docs should emphasize that master and slave configurations should
> hostnames wherever possible.
> Link to current docs:
> https://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-hdfs/HdfsMultihoming.html
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-8891) HDFS concat should keep srcs order
Yong Zhang created HDFS-8891: Summary: HDFS concat should keep srcs order Key: HDFS-8891 URL: https://issues.apache.org/jira/browse/HDFS-8891 Project: Hadoop HDFS Issue Type: Improvement Reporter: Yong Zhang Assignee: Yong Zhang FSDirConcatOp.verifySrcFiles may change src files order, but it should their order as input. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8622) Implement GETCONTENTSUMMARY operation for WebImageViewer
[
https://issues.apache.org/jira/browse/HDFS-8622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14694728#comment-14694728
]
Hudson commented on HDFS-8622:
--
FAILURE: Integrated in Hadoop-trunk-Commit #8292 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/8292/])
HDFS-8622. Implement GETCONTENTSUMMARY operation for WebImageViewer.
Contributed by Jagadesh Kiran N. (aajisaka: rev
40f815131e822f5b7a8e6a6827f4b85b31220c43)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/FSImageHandler.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/TestOfflineImageViewerForContentSummary.java
* hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsImageViewer.md
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/FSImageLoader.java
> Implement GETCONTENTSUMMARY operation for WebImageViewer
>
>
> Key: HDFS-8622
> URL: https://issues.apache.org/jira/browse/HDFS-8622
> Project: Hadoop HDFS
> Issue Type: New Feature
>Reporter: Jagadesh Kiran N
>Assignee: Jagadesh Kiran N
> Attachments: HDFS-8622-00.patch, HDFS-8622-01.patch,
> HDFS-8622-02.patch, HDFS-8622-03.patch, HDFS-8622-04.patch,
> HDFS-8622-05.patch, HDFS-8622-06.patch, HDFS-8622-07.patch,
> HDFS-8622-08.patch, HDFS-8622-09.patch, HDFS-8622-10.patch
>
>
> it would be better for administrators if {code} GETCONTENTSUMMARY {code} are
> supported.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8854) Erasure coding: add ECPolicy to replace schema+cellSize in hadoop-hdfs
[ https://issues.apache.org/jira/browse/HDFS-8854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14694758#comment-14694758 ] Zhe Zhang commented on HDFS-8854: - Will try, thanks! > Erasure coding: add ECPolicy to replace schema+cellSize in hadoop-hdfs > -- > > Key: HDFS-8854 > URL: https://issues.apache.org/jira/browse/HDFS-8854 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7285 >Reporter: Walter Su >Assignee: Walter Su > Attachments: HDFS-8854-Consolidated-20150806.02.txt, > HDFS-8854-HDFS-7285-merge.03.patch, HDFS-8854-HDFS-7285-merge.03.txt, > HDFS-8854-HDFS-7285.00.patch, HDFS-8854-HDFS-7285.01.patch, > HDFS-8854-HDFS-7285.02.patch, HDFS-8854-HDFS-7285.03.patch, HDFS-8854.00.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8854) Erasure coding: add ECPolicy to replace schema+cellSize in hadoop-hdfs
[ https://issues.apache.org/jira/browse/HDFS-8854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14694760#comment-14694760 ] Zhe Zhang commented on HDFS-8854: - Jenkins was not stable and [reported | https://builds.apache.org/job/Hadoop-HDFS-7285-Merge/83/] many unrelated failures. Will wait for another run. > Erasure coding: add ECPolicy to replace schema+cellSize in hadoop-hdfs > -- > > Key: HDFS-8854 > URL: https://issues.apache.org/jira/browse/HDFS-8854 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7285 >Reporter: Walter Su >Assignee: Walter Su > Attachments: HDFS-8854-Consolidated-20150806.02.txt, > HDFS-8854-HDFS-7285-merge.03.patch, HDFS-8854-HDFS-7285-merge.03.txt, > HDFS-8854-HDFS-7285.00.patch, HDFS-8854-HDFS-7285.01.patch, > HDFS-8854-HDFS-7285.02.patch, HDFS-8854-HDFS-7285.03.patch, HDFS-8854.00.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8808) dfs.image.transfer.bandwidthPerSec should not apply to -bootstrapStandby
[
https://issues.apache.org/jira/browse/HDFS-8808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang updated HDFS-8808:
Attachment: HDFS-8808-03.patch
Updating patch to fix whitespace issue (it was actually not caused by the
patch, fixing anyway).
{{TestReplaceDatanodeOnFailure}} is unrelated to the change and passes locally.
> dfs.image.transfer.bandwidthPerSec should not apply to -bootstrapStandby
>
>
> Key: HDFS-8808
> URL: https://issues.apache.org/jira/browse/HDFS-8808
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.1
>Reporter: Gautam Gopalakrishnan
>Assignee: Zhe Zhang
> Attachments: HDFS-8808-00.patch, HDFS-8808-01.patch,
> HDFS-8808-02.patch, HDFS-8808-03.patch
>
>
> The parameter {{dfs.image.transfer.bandwidthPerSec}} can be used to limit the
> speed with which the fsimage is copied between the namenodes during regular
> use. However, as a side effect, this also limits transfers when the
> {{-bootstrapStandby}} option is used. This option is often used during
> upgrades and could potentially slow down the entire workflow. The request
> here is to ensure {{-bootstrapStandby}} is unaffected by this bandwidth
> setting
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7831) Fix the starting index and end condition of the loop in FileDiffList.findEarlierSnapshotBlocks()
[
https://issues.apache.org/jira/browse/HDFS-7831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sangjin Lee updated HDFS-7831:
--
Labels: (was: 2.6.1-candidate)
Removing the 2.6.1-candidate label as this is not applicable to 2.6.
> Fix the starting index and end condition of the loop in
> FileDiffList.findEarlierSnapshotBlocks()
>
>
> Key: HDFS-7831
> URL: https://issues.apache.org/jira/browse/HDFS-7831
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.0
>Reporter: Konstantin Shvachko
>Assignee: Konstantin Shvachko
> Fix For: 2.7.0
>
> Attachments: HDFS-7831-01.patch
>
>
> Currently the loop in {{FileDiffList.findEarlierSnapshotBlocks()}} starts
> from {{insertPoint + 1}}. It should start from {{insertPoint - 1}}. As noted
> in [Jing's
> comment|https://issues.apache.org/jira/browse/HDFS-7056?focusedCommentId=14333864&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14333864]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7843) A truncated file is corrupted after rollback from a rolling upgrade
[ https://issues.apache.org/jira/browse/HDFS-7843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sangjin Lee updated HDFS-7843: -- Labels: (was: 2.6.1-candidate) Removing the 2.6.1-candidate label as this is not applicable to 2.6.0. > A truncated file is corrupted after rollback from a rolling upgrade > --- > > Key: HDFS-7843 > URL: https://issues.apache.org/jira/browse/HDFS-7843 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze >Priority: Blocker > Fix For: 2.7.0 > > Attachments: h7843_20150226.patch > > > Here is a rolling upgrade truncate test from [~brandonli]. The basic test > step is: (3 nodes cluster with HA) > 1. upload a file to hdfs > 2. start rollingupgrade. finish rollingupgrade for namenode and one datanode. > 3. truncate the file in hdfs to 1byte > 4. do rollback > 5. download file from hdfs, check file size to be original size > I see the file size in hdfs is correct but can't read it because the block is > corrupted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
