[jira] [Commented] (HDFS-9796) Add throttler for datanode bandwidth
[
https://issues.apache.org/jira/browse/HDFS-9796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15146417#comment-15146417
]
Hadoop QA commented on HDFS-9796:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
37s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 52s
{color} | {color:green} trunk passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 42s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 52s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
59s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 13s
{color} | {color:green} trunk passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 54s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
59s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 2s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 2s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 47s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 47s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 24s
{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs: patch generated 2 new +
464 unchanged - 0 fixed = 466 total (was 464) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 56s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 38s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 41s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 98m 49s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_72. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 74m 30s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_95. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 203m 47s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_72 Failed junit tests |
hadoop.hdfs.server.datanode.TestBlockScanner |
| | hadoop.hdfs.server.datanode.TestFsDatasetCache |
| | hadoop.hdfs.server.datanode.TestDataNodeMetrics |
| | hadoop.hdfs.TestFileAppend |
| | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
| | hadoop.hdfs.TestSafeMode |
| | hadoop.hdfs.server.namenode.TestDecommissioningStatus
[jira] [Commented] (HDFS-9780) RollingFileSystemSink doesn't work on secure clusters
[
https://issues.apache.org/jira/browse/HDFS-9780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15146382#comment-15146382
]
Daniel Templeton commented on HDFS-9780:
The issue is not this patch. You're seeing HADOOP-12090 rearing its head. The
message happened to be split across two TCP packets, causing the KDC to reject
it. If you look at the stack trace in the HDFS-9711 logs, it's identical to
HADOOP-12090:
{code}
EXCEPTION
org.apache.mina.filter.codec.ProtocolDecoderException:
java.lang.NullPointerException: message (Hexdump: 00 00 02 43 6C 82 02 3F 30 82
02 3B A1 03 02 01 05 A2 03 02 01 0C A3 82 01 C2 30 82 01 BE 30 82 01 BA A1 03
02 01 01 A2 82 01 B1 04 82 01 AD 6E 82 01 A9 30 82 01 A5 A0 03 02 01 05 A1 03
02 01 0E A2 07 03 05 00 00 00 00 00 A3 81 F5 61 81 F2 30 81 EF A0 03 02 01 05
A1 0D 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A2 20 30 1E A0 03 02 01 02 A1 17
30 15 1B 06 6B 72 62 74 67 74 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A3 81 B6
30 81 B3 A0 03 02 01 11 A2 81 AB 04 81 A8 CD 9D D5 D9 84 E1 75 0B AE 35 44 84
E6 25 3D 07 9F 15 43 E8 31 D3 8D 2A 45 41 F2 2B B4 4D C4 0F 24 A2 70 F2 53 2A
C7 A9 9C 93 54 43 6E C0 5F B4 47 49 7F 49 C1 96 18 4E F1 F6 07 55 A7 B2 2C 6C
6A D1 82 60 A1 04 29 D2 BB 38 6D D9 88 40 7B 85 24 CD B3 37 04 08 34 4D C4 24
EB 45 CE AD 70 6C 3A C2 8E 73 60 6F D4 36 44 D0 01 62 75 73 85 91 98 54 59 05
E8 B4 34 72 00 30 D5 4F 4C 39 4D D9 4C 5E 46 C6 63 DB 6D B0 15 D6 88 0A F0 C5
C8 43 A7 9D 71 69 AF 80 4A 52 DB 8D 0B E0 C0 80 24 28 EF DC F4 71 B5 2F 91 64
A4 81 97 30 81 94 A0 03 02 01 11 A2 81 8C 04 81 89 17 CE ED C3 54 CD A3 A0 81
CE 3C 31 B4 09 2C 29 A2 B4 30 5D 88 28 04 75 15 CA D7 3C 0C B2 DF 07 84 A4 0D
AF 71 EB 55 39 D4 A1 C0 70 99 9A 11 9E DA C3 96 ED C2 6F 3E 32 F8 55 83 0C F8
3E 8E 05 20 46 49 02 AF 0C 6A 53 58 80 93 F4 14 86 A9 60 2E 00 CD BD 6E 93 23
9D 8B 5C D9 D8 96 23 56 AB 61 7F 78 29 29 98 86 33 48 A2 11 F7 6F 11 39 CE D4
FB F4 DE 76 88 70 A7 09 8C CA A8 BC C6 89 A6 7D 7C FB 50 EA 47 7B 43 7F A4 69
30 67 A0 07 03 05 00 00 00 00 00 A2 0D 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D
A3 1C 30 1A A0 03 02 01 00 A1 13 30 11 1B 04 68 64 66 73 1B 09 6C 6F 63 61 6C
68 6F 73 74 A5 11 18 0F 31 39 37 30 30 31 30 31 30 30 30 30 30 30 5A A7 06 02
04 4E 5A 84 B4 A8 14 30 12 02 01 12 02 01 11 02 01 10 02 01)
at
org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:234)
at
org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434)
at
org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:48)
at
org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:802)
at
org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:120)
at
org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434)
at
org.apache.mina.core.filterchain.DefaultIoFilterChain.fireMessageReceived(DefaultIoFilterChain.java:426)
at
org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:604)
at
org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:564)
at
org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:553)
at
org.apache.mina.core.polling.AbstractPollingIoProcessor.access$400(AbstractPollingIoProcessor.java:57)
at
org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:892)
at
org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:65)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException: message
at
org.apache.mina.filter.codec.AbstractProtocolDecoderOutput.write(AbstractProtocolDecoderOutput.java:44)
at
org.apache.directory.server.kerberos.protocol.codec.MinaKerberosDecoder.decode(MinaKerberosDecoder.java:65)
at
org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:224)
... 15 more
{code}
> RollingFileSystemSink doesn't work on secure clusters
> -
>
> Key: HDFS-9780
> URL: https://issues.apache.org/jira/browse/HDFS-9780
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.9.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Critical
> Attachments: HADOOP-12775.001.patch, HADOOP-12775.002.patch,
>
[jira] [Assigned] (HDFS-9796) Add throttler for datanode bandwidth
[
https://issues.apache.org/jira/browse/HDFS-9796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guocui Mi reassigned HDFS-9796:
---
Assignee: Guocui Mi
> Add throttler for datanode bandwidth
>
>
> Key: HDFS-9796
> URL: https://issues.apache.org/jira/browse/HDFS-9796
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode
>Affects Versions: 2.7.0
>Reporter: Guocui Mi
>Assignee: Guocui Mi
>Priority: Minor
> Attachments: HDFS-9796-1.patch
>
>
> Add throttler for datanode bandwidth in {{DataXceiver}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9796) Add throttler for datanode bandwidth
[
https://issues.apache.org/jira/browse/HDFS-9796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guocui Mi updated HDFS-9796:
Status: Patch Available (was: Open)
> Add throttler for datanode bandwidth
>
>
> Key: HDFS-9796
> URL: https://issues.apache.org/jira/browse/HDFS-9796
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode
>Affects Versions: 2.7.0
>Reporter: Guocui Mi
>Assignee: Guocui Mi
>Priority: Minor
> Attachments: HDFS-9796-1.patch
>
>
> Add throttler for datanode bandwidth in {{DataXceiver}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9796) Add throttler for datanode bandwidth
[
https://issues.apache.org/jira/browse/HDFS-9796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guocui Mi updated HDFS-9796:
Attachment: HDFS-9796-1.patch
> Add throttler for datanode bandwidth
>
>
> Key: HDFS-9796
> URL: https://issues.apache.org/jira/browse/HDFS-9796
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode
>Affects Versions: 2.7.0
>Reporter: Guocui Mi
>Priority: Minor
> Attachments: HDFS-9796-1.patch
>
>
> Add throttler for datanode bandwidth in {{DataXceiver}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9796) Add throttler for datanode bandwidth
[
https://issues.apache.org/jira/browse/HDFS-9796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15146343#comment-15146343
]
Guocui Mi commented on HDFS-9796:
-
Yes, this is what i want to have.
> Add throttler for datanode bandwidth
>
>
> Key: HDFS-9796
> URL: https://issues.apache.org/jira/browse/HDFS-9796
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode
>Affects Versions: 2.7.0
>Reporter: Guocui Mi
>Priority: Minor
>
> Add throttler for datanode bandwidth in {{DataXceiver}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9796) Add throttler for datanode bandwidth
[
https://issues.apache.org/jira/browse/HDFS-9796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15146341#comment-15146341
]
Guocui Mi commented on HDFS-9796:
-
Sorry for delay response, thanks Yiqun and Inigo. @Yiqun, i have a simple patch
in my local, i can upload the patch shortly, could you help review?
> Add throttler for datanode bandwidth
>
>
> Key: HDFS-9796
> URL: https://issues.apache.org/jira/browse/HDFS-9796
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode
>Affects Versions: 2.7.0
>Reporter: Guocui Mi
>Priority: Minor
>
> Add throttler for datanode bandwidth in {{DataXceiver}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9413) getContentSummary() on standby should throw StandbyException
[ https://issues.apache.org/jira/browse/HDFS-9413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15146323#comment-15146323 ] caixiaofeng commented on HDFS-9413: --- 马克 > getContentSummary() on standby should throw StandbyException > > > Key: HDFS-9413 > URL: https://issues.apache.org/jira/browse/HDFS-9413 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula >Priority: Critical > Fix For: 2.7.2 > > Attachments: HDFS-9413-002.patch, HDFS-9413-003.patch, HDFS-9413.patch > > > Currently when we call getContentSummary() on standby it will not throw > StandbyException. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9796) Add throttler for datanode bandwidth
[
https://issues.apache.org/jira/browse/HDFS-9796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15146303#comment-15146303
]
Lin Yiqun commented on HDFS-9796:
-
It's a good idea for to do this. If there are some heterogeneous machines in
the cluster, different node has different bandwidth and we can add throttler to
control the io rate. So [~elgoiri], do we need do it now? I'm glad to work for
this if this jira still not be assigned.
> Add throttler for datanode bandwidth
>
>
> Key: HDFS-9796
> URL: https://issues.apache.org/jira/browse/HDFS-9796
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode
>Affects Versions: 2.7.0
>Reporter: Guocui Mi
>Priority: Minor
>
> Add throttler for datanode bandwidth in {{DataXceiver}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9711) Integrate CSRF prevention filter in WebHDFS.
[
https://issues.apache.org/jira/browse/HDFS-9711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15146225#comment-15146225
]
Chris Nauroth commented on HDFS-9711:
-
I can see that {{TestRollingFileSystemSinkWithSecureHdfs}} fails when run on
JDK 7, but not when run on JDK 8. This is unrelated to the patch. This test
was introduced in HDFS-9780, so I've asked the contributors on that one to take
a look.
Aside from that, I cannot reproduce the other test failures.
The Checkstyle warning is about a missing package-info.java file for the
{{org.apache.hadoop.hdfs.server.datanode.web}} package. It's not worthwhile
adding this for a package that isn't part of the public API.
> Integrate CSRF prevention filter in WebHDFS.
>
>
> Key: HDFS-9711
> URL: https://issues.apache.org/jira/browse/HDFS-9711
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, namenode, webhdfs
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
> Attachments: HDFS-9711.001.patch, HDFS-9711.002.patch,
> HDFS-9711.003.patch, HDFS-9711.004.patch, HDFS-9711.005.patch
>
>
> HADOOP-12691 introduced a filter in Hadoop Common to help REST APIs guard
> against cross-site request forgery attacks. This issue tracks integration of
> that filter in WebHDFS.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9780) RollingFileSystemSink doesn't work on secure clusters
[
https://issues.apache.org/jira/browse/HDFS-9780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15146224#comment-15146224
]
Chris Nauroth commented on HDFS-9780:
-
I am seeing {{TestRollingFileSystemSinkWithHdfs}} fail with Kerberos errors
when running on JDK 7. It seems to pass fine on JDK 8. See HDFS-9711 for an
example of it failing in pre-commit. Shall we revert this, or would one of the
original contributors like to take a look first?
> RollingFileSystemSink doesn't work on secure clusters
> -
>
> Key: HDFS-9780
> URL: https://issues.apache.org/jira/browse/HDFS-9780
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.9.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Critical
> Attachments: HADOOP-12775.001.patch, HADOOP-12775.002.patch,
> HADOOP-12775.003.patch, HDFS-9780.004.patch, HDFS-9780.005.patch,
> HDFS-9780.006.patch, HDFS-9780.006.patch, HDFS-9780.007.patch,
> HDFS-9780.008.patch
>
>
> If HDFS has kerberos enabled, the sink cannot write its logs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7694) FSDataInputStream should support "unbuffer"
[
https://issues.apache.org/jira/browse/HDFS-7694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15146220#comment-15146220
]
Chris Nauroth commented on HDFS-7694:
-
As per discussion on HADOOP-12805 with the HBase community, I'd like to make
{{CanUnbuffer}} a {{Public}} interface. I'd appreciate if the original
contributors here could help chime in on HADOOP-12805. Thanks!
> FSDataInputStream should support "unbuffer"
> ---
>
> Key: HDFS-7694
> URL: https://issues.apache.org/jira/browse/HDFS-7694
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 2.7.0
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Fix For: 2.7.0, 2.6.4
>
> Attachments: HDFS-7694.001.patch, HDFS-7694.002.patch,
> HDFS-7694.003.patch, HDFS-7694.004.patch, HDFS-7694.005.patch
>
>
> For applications that have many open HDFS (or other Hadoop filesystem) files,
> it would be useful to have an API to clear readahead buffers and sockets.
> This could be added to the existing APIs as an optional interface, in much
> the same way as we added setReadahead / setDropBehind / etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9810) Allow support for more than one block replica per datanode
[ https://issues.apache.org/jira/browse/HDFS-9810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15146194#comment-15146194 ] Virajith Jalaparti commented on HDFS-9810: -- Our use case is driven by HDFS-9806 -- the idea is when a "provided" block is fetched by a datanode, the datanode can cache it on local storage media (disk, SSD etc.) so that future reads of the data are served from it. When the block is cached, the datanode has to maintain information for two replicas (in FSDatasetImpl), one for the local replica and the other for the "provided" replica. With this, if the local replica is lost, it can still be accessed from the provided replica. This is a modification in the FSDatasetImpl to enable the possibility of a datanode storing multiple replicas for a block. We are not proposing to change the BlockPlacementPolicyDefault, which determines how the replicas are placed. The BlockPlacementPolicyDefault will still work as it works today and hence, the failure model will remain the same. > Allow support for more than one block replica per datanode > -- > > Key: HDFS-9810 > URL: https://issues.apache.org/jira/browse/HDFS-9810 > Project: Hadoop HDFS > Issue Type: Task > Components: datanode >Reporter: Virajith Jalaparti > > Datanodes report and store only one replica of each block. It should be > possible to store multiple replicas among its different configured storage > types, particularly to support non-durable media and remote storage. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9796) Add throttler for datanode bandwidth
[
https://issues.apache.org/jira/browse/HDFS-9796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Inigo Goiri updated HDFS-9796:
--
Description: Add throttler for datanode bandwidth in {{}} (was: Add
throttler for datanode bandwidth)
> Add throttler for datanode bandwidth
>
>
> Key: HDFS-9796
> URL: https://issues.apache.org/jira/browse/HDFS-9796
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode
>Affects Versions: 2.7.0
>Reporter: Guocui Mi
>Priority: Minor
>
> Add throttler for datanode bandwidth in {{}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9796) Add throttler for datanode bandwidth
[
https://issues.apache.org/jira/browse/HDFS-9796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15146153#comment-15146153
]
Inigo Goiri commented on HDFS-9796:
---
The proposal is to add {{DataTransferThrottler}} (already used by
{{BlockBalanceThrottler}}) to {{DataXceiverServer}} and then using it in
{{DataXceiver#readBlock}} and {{DataXceiver#writeBlock}} as [~linyiqun] pointed
out. This setting would have a config setting to enable it or not.
> Add throttler for datanode bandwidth
>
>
> Key: HDFS-9796
> URL: https://issues.apache.org/jira/browse/HDFS-9796
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode
>Affects Versions: 2.7.0
>Reporter: Guocui Mi
>Priority: Minor
>
> Add throttler for datanode bandwidth
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9796) Add throttler for datanode bandwidth
[
https://issues.apache.org/jira/browse/HDFS-9796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Inigo Goiri updated HDFS-9796:
--
Description: Add throttler for datanode bandwidth in {{DataXceiver}} (was:
Add throttler for datanode bandwidth in {{}})
> Add throttler for datanode bandwidth
>
>
> Key: HDFS-9796
> URL: https://issues.apache.org/jira/browse/HDFS-9796
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode
>Affects Versions: 2.7.0
>Reporter: Guocui Mi
>Priority: Minor
>
> Add throttler for datanode bandwidth in {{DataXceiver}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9521) TransferFsImage.receiveFile should account and log separate times for image download and fsync to disk
[
https://issues.apache.org/jira/browse/HDFS-9521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15146118#comment-15146118
]
Hadoop QA commented on HDFS-9521:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 19s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
9s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 50s
{color} | {color:green} trunk passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 42s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 54s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
57s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 13s
{color} | {color:green} trunk passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 59s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 50s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 50s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 41s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 41s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 52s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 9s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 14s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 50s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 77m 33s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_72. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 87m 16s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_95. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 192m 3s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_72 Failed junit tests |
hadoop.hdfs.server.datanode.TestBlockScanner |
| | hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency |
| | hadoop.hdfs.TestErasureCodeBenchmarkThroughput |
| | hadoop.hdfs.security.TestDelegationTokenForProxyUser |
| | hadoop.hdfs.server.mover.TestStorageMover |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery
|
| JDK v1.7.0_95 Failed junit tests | hadoop.hdfs.TestDFSUpgradeFromImage |
| | hadoop.hdfs.server.namenode.web.resources.TestWebHdfsD
[jira] [Updated] (HDFS-9521) TransferFsImage.receiveFile should account and log separate times for image download and fsync to disk
[
https://issues.apache.org/jira/browse/HDFS-9521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wellington Chevreuil updated HDFS-9521:
---
Status: In Progress (was: Patch Available)
> TransferFsImage.receiveFile should account and log separate times for image
> download and fsync to disk
> ---
>
> Key: HDFS-9521
> URL: https://issues.apache.org/jira/browse/HDFS-9521
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Wellington Chevreuil
>Assignee: Wellington Chevreuil
>Priority: Minor
> Attachments: HDFS-9521-2.patch, HDFS-9521.patch, HDFS-9521.patch.1
>
>
> Currently, TransferFsImage.receiveFile is logging total transfer time as
> below:
> {noformat}
> double xferSec = Math.max(
>((float)(Time.monotonicNow() - startTime)) / 1000.0, 0.001);
> long xferKb = received / 1024;
> LOG.info(String.format("Transfer took %.2fs at %.2f KB/s",xferSec, xferKb /
> xferSec))
> {noformat}
> This is really useful, but it just measures the total method execution time,
> which includes time taken to download the image and do an fsync to all the
> namenode metadata directories.
> Sometime when troubleshooting these imager transfer problems, it's
> interesting to know which part of the process is being the bottleneck
> (whether network or disk write).
> This patch accounts time for image download and fsync to each disk
> separately, logging how much time did it take on each operation.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9521) TransferFsImage.receiveFile should account and log separate times for image download and fsync to disk
[
https://issues.apache.org/jira/browse/HDFS-9521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wellington Chevreuil updated HDFS-9521:
---
Status: Patch Available (was: In Progress)
> TransferFsImage.receiveFile should account and log separate times for image
> download and fsync to disk
> ---
>
> Key: HDFS-9521
> URL: https://issues.apache.org/jira/browse/HDFS-9521
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Wellington Chevreuil
>Assignee: Wellington Chevreuil
>Priority: Minor
> Attachments: HDFS-9521-2.patch, HDFS-9521.patch, HDFS-9521.patch.1
>
>
> Currently, TransferFsImage.receiveFile is logging total transfer time as
> below:
> {noformat}
> double xferSec = Math.max(
>((float)(Time.monotonicNow() - startTime)) / 1000.0, 0.001);
> long xferKb = received / 1024;
> LOG.info(String.format("Transfer took %.2fs at %.2f KB/s",xferSec, xferKb /
> xferSec))
> {noformat}
> This is really useful, but it just measures the total method execution time,
> which includes time taken to download the image and do an fsync to all the
> namenode metadata directories.
> Sometime when troubleshooting these imager transfer problems, it's
> interesting to know which part of the process is being the bottleneck
> (whether network or disk write).
> This patch accounts time for image download and fsync to each disk
> separately, logging how much time did it take on each operation.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9521) TransferFsImage.receiveFile should account and log separate times for image download and fsync to disk
[
https://issues.apache.org/jira/browse/HDFS-9521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wellington Chevreuil updated HDFS-9521:
---
Attachment: HDFS-9521-2.patch
New patch version. Combined all logs in a single log version.
> TransferFsImage.receiveFile should account and log separate times for image
> download and fsync to disk
> ---
>
> Key: HDFS-9521
> URL: https://issues.apache.org/jira/browse/HDFS-9521
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Wellington Chevreuil
>Assignee: Wellington Chevreuil
>Priority: Minor
> Attachments: HDFS-9521-2.patch, HDFS-9521.patch, HDFS-9521.patch.1
>
>
> Currently, TransferFsImage.receiveFile is logging total transfer time as
> below:
> {noformat}
> double xferSec = Math.max(
>((float)(Time.monotonicNow() - startTime)) / 1000.0, 0.001);
> long xferKb = received / 1024;
> LOG.info(String.format("Transfer took %.2fs at %.2f KB/s",xferSec, xferKb /
> xferSec))
> {noformat}
> This is really useful, but it just measures the total method execution time,
> which includes time taken to download the image and do an fsync to all the
> namenode metadata directories.
> Sometime when troubleshooting these imager transfer problems, it's
> interesting to know which part of the process is being the bottleneck
> (whether network or disk write).
> This patch accounts time for image download and fsync to each disk
> separately, logging how much time did it take on each operation.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9796) Add throttler for datanode bandwidth
[
https://issues.apache.org/jira/browse/HDFS-9796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15145883#comment-15145883
]
Lin Yiqun commented on HDFS-9796:
-
Is it means to add the throttler in {{DataXceiver#readBlock}} and
{{DataXceiver#writeBlock}} and control the datanode io rate? And now these two
methods have not limited the bandwidth.
> Add throttler for datanode bandwidth
>
>
> Key: HDFS-9796
> URL: https://issues.apache.org/jira/browse/HDFS-9796
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode
>Affects Versions: 2.7.0
>Reporter: Guocui Mi
>Priority: Minor
>
> Add throttler for datanode bandwidth
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9671) DiskBalancer : SubmitPlan implementation
[
https://issues.apache.org/jira/browse/HDFS-9671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15145875#comment-15145875
]
Hadoop QA commented on HDFS-9671:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 30s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 43s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m
19s {color} | {color:green} HDFS-1312 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 42s
{color} | {color:green} HDFS-1312 passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 40s
{color} | {color:green} HDFS-1312 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s {color} | {color:green} HDFS-1312 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 56s
{color} | {color:green} HDFS-1312 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
35s {color} | {color:green} HDFS-1312 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
59s {color} | {color:green} HDFS-1312 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 59s
{color} | {color:green} HDFS-1312 passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 45s
{color} | {color:green} HDFS-1312 passed with JDK v1.7.0_95 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
38s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 1s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 1s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 42s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 42s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 34s
{color} | {color:red} hadoop-hdfs-project: patch generated 9 new + 515
unchanged - 1 fixed = 524 total (was 516) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 48s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
32s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m 8s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 40s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 14s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 32s
{color} | {color:green} hadoop-hdfs-client in the patch passed with JDK
v1.8.0_72. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 114m 29s
{color} | {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_72.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 17s
{color} | {color:green} hadoop-hdfs-client in the patch passed with JDK
v1.7.0_95. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 91m 13s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_95. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 29s
{color} | {color:red} Patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 264m 15s {c
[jira] [Commented] (HDFS-9772) TestBlockReplacement#testThrottler use falut variable to calculate bandwidth
[
https://issues.apache.org/jira/browse/HDFS-9772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15145872#comment-15145872
]
Lin Yiqun commented on HDFS-9772:
-
[~acmurthy], could you please also see this patch, the comparsion code
{{totalBytes*1000/(end-start)<=bandwidthPerSec}} is committed by you, thanks.
> TestBlockReplacement#testThrottler use falut variable to calculate bandwidth
>
>
> Key: HDFS-9772
> URL: https://issues.apache.org/jira/browse/HDFS-9772
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.1
>Reporter: Lin Yiqun
>Assignee: Lin Yiqun
> Attachments: HDFS.001.patch
>
>
> In {{TestBlockReplacement#testThrottler}}, it use a fault variable to
> calculate the ended bandwidth. It use variable {{totalBytes}} rathe than
> final variable {{TOTAL_BYTES}}. And the value of {{TOTAL_BYTES}} is set to
> {{bytesToSend}}. The {{totalBytes}} looks no meaning here and this will make
> {{totalBytes*1000/(end-start)}} always be 0 and the comparison always true.
> The method code is below:
> {code}
> @Test
> public void testThrottler() throws IOException {
> Configuration conf = new HdfsConfiguration();
> FileSystem.setDefaultUri(conf, "hdfs://localhost:0");
> long bandwidthPerSec = 1024*1024L;
> final long TOTAL_BYTES =6*bandwidthPerSec;
> long bytesToSend = TOTAL_BYTES;
> long start = Time.monotonicNow();
> DataTransferThrottler throttler = new
> DataTransferThrottler(bandwidthPerSec);
> long totalBytes = 0L;
> long bytesSent = 1024*512L; // 0.5MB
> throttler.throttle(bytesSent);
> bytesToSend -= bytesSent;
> bytesSent = 1024*768L; // 0.75MB
> throttler.throttle(bytesSent);
> bytesToSend -= bytesSent;
> try {
> Thread.sleep(1000);
> } catch (InterruptedException ignored) {}
> throttler.throttle(bytesToSend);
> long end = Time.monotonicNow();
> assertTrue(totalBytes*1000/(end-start)<=bandwidthPerSec);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
