[jira] [Created] (HDFS-14310) Improve documents for using DNS to resolve namenodes and routers
Fengnan Li created HDFS-14310: - Summary: Improve documents for using DNS to resolve namenodes and routers Key: HDFS-14310 URL: https://issues.apache.org/jira/browse/HDFS-14310 Project: Hadoop HDFS Issue Type: Improvement Reporter: Fengnan Li Assignee: Fengnan Li With https://issues.apache.org/jira/browse/HDFS-14118, clients can just use one domain name to access either namenodes or routers, instead of putting all of their domain names in the config. Update the below documents with this new feature: * hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSHighAvailabilityWithQJM.md * hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSHighAvailabilityWithNFS.md * hadoop-hdfs-project/hadoop-hdfs-rbf/src/site/markdown/HDFSRouterFederation.md * hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ObserverNamenode.md -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14118) Use DNS to resolve Namenodes and Routers
[ https://issues.apache.org/jira/browse/HDFS-14118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16774868#comment-16774868 ] Fengnan Li commented on HDFS-14118: --- [~yzhangal] Thanks for the review! I removed the statement about "Random ..." in hdfs-default.xml and also update the dfs.client.failover.random.order a little with detailed scenarios. HDFS-14310 is linked to this one and I will work on docs there. > Use DNS to resolve Namenodes and Routers > > > Key: HDFS-14118 > URL: https://issues.apache.org/jira/browse/HDFS-14118 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Fengnan Li >Assignee: Fengnan Li >Priority: Major > Attachments: DNS testing log, HDFS design doc_ Single domain name for > clients - Google Docs-1.pdf, HDFS design doc_ Single domain name for clients > - Google Docs.pdf, HDFS-14118.001.patch, HDFS-14118.002.patch, > HDFS-14118.003.patch, HDFS-14118.004.patch, HDFS-14118.005.patch, > HDFS-14118.006.patch, HDFS-14118.007.patch, HDFS-14118.008.patch, > HDFS-14118.009.patch, HDFS-14118.010.patch, HDFS-14118.011.patch, > HDFS-14118.012.patch, HDFS-14118.013.patch, HDFS-14118.014.patch, > HDFS-14118.015.patch, HDFS-14118.016.patch, HDFS-14118.017.patch, > HDFS-14118.018.patch, HDFS-14118.019.patch, HDFS-14118.020.patch, > HDFS-14118.021.patch, HDFS-14118.patch > > > Clients will need to know about routers to talk to the HDFS cluster > (obviously), and having routers updating (adding/removing) will have to make > every client change, which is a painful process. > DNS can be used here to resolve the single domain name clients knows to a > list of routers in the current config. However, DNS won't be able to consider > only resolving to the working router based on certain health thresholds. > There are some ways about how this can be solved. One way is to have a > separate script to regularly check the status of the router and update the > DNS records if a router fails the health thresholds. In this way, security > might be carefully considered for this way. Another way is to have the client > do the normal connecting/failover after they get the list of routers, which > requires the change of current failover proxy provider. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-14311) multi-threading conflict at layoutVersion when loading block pool storage
Yicong Cai created HDFS-14311:
-
Summary: multi-threading conflict at layoutVersion when loading
block pool storage
Key: HDFS-14311
URL: https://issues.apache.org/jira/browse/HDFS-14311
Project: Hadoop HDFS
Issue Type: Bug
Components: rolling upgrades
Affects Versions: 2.9.2
Reporter: Yicong Cai
When DataNode upgrade from 2.7.3 to 2.9.2, there is a conflict at
StorageInfo.layoutVersion in loading block pool storage process.
It will cause this exception:
{panel:title=exceptions}
2019-02-15 10:18:01,357 [13783] - INFO [Thread-33:BlockPoolSliceStorage@395] -
Restored 36974 block files from trash before the layout upgrade. These blocks
will be moved to the previous directory during the upgrade
2019-02-15 10:18:01,358 [13784] - WARN [Thread-33:BlockPoolSliceStorage@226] -
Failed to analyze storage directories for block pool
BP-1216718839-10.120.232.23-1548736842023
java.io.IOException: Datanode state: LV = -57 CTime = 0 is newer than the
namespace state: LV = -63 CTime = 0
at
org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceStorage.doTransition(BlockPoolSliceStorage.java:406)
at
org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceStorage.loadStorageDirectory(BlockPoolSliceStorage.java:177)
at
org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceStorage.loadBpStorageDirectories(BlockPoolSliceStorage.java:221)
at
org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceStorage.recoverTransitionRead(BlockPoolSliceStorage.java:250)
at
org.apache.hadoop.hdfs.server.datanode.DataStorage.loadBlockPoolSliceStorage(DataStorage.java:460)
at
org.apache.hadoop.hdfs.server.datanode.DataStorage.addStorageLocations(DataStorage.java:390)
at
org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:556)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:1649)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1610)
at
org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:388)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:280)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:816)
at java.lang.Thread.run(Thread.java:748)
2019-02-15 10:18:01,358 [13784] - WARN [Thread-33:DataStorage@472] - Failed to
add storage directory [DISK]file:/mnt/dfs/2/hadoop/hdfs/data/ for block pool
BP-1216718839-10.120.232.23-1548736842023
java.io.IOException: Datanode state: LV = -57 CTime = 0 is newer than the
namespace state: LV = -63 CTime = 0
at
org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceStorage.doTransition(BlockPoolSliceStorage.java:406)
at
org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceStorage.loadStorageDirectory(BlockPoolSliceStorage.java:177)
at
org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceStorage.loadBpStorageDirectories(BlockPoolSliceStorage.java:221)
at
org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceStorage.recoverTransitionRead(BlockPoolSliceStorage.java:250)
at
org.apache.hadoop.hdfs.server.datanode.DataStorage.loadBlockPoolSliceStorage(DataStorage.java:460)
at
org.apache.hadoop.hdfs.server.datanode.DataStorage.addStorageLocations(DataStorage.java:390)
at
org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:556)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:1649)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1610)
at
org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:388)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:280)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:816)
at java.lang.Thread.run(Thread.java:748)
{panel}
root cause:
BlockPoolSliceStorage instance is shared for all storage locations recover
transition. In BlockPoolSliceStorage.doTransition, it will read the old
layoutVersion from local storage, compare with current DataNode version, then
do upgrade. In doUpgrade, add the transition work as a sub-thread, the
transition work will set the BlockPoolSliceStorage's layoutVersion to current
DN version. The next storage dir transition check will concurrent with pre
storage dir real transition work, then the BlockPoolSliceStorage instance
layoutVersion will confusion.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14118) Use DNS to resolve Namenodes and Routers
[ https://issues.apache.org/jira/browse/HDFS-14118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fengnan Li updated HDFS-14118: -- Attachment: HDFS-14118.021.patch > Use DNS to resolve Namenodes and Routers > > > Key: HDFS-14118 > URL: https://issues.apache.org/jira/browse/HDFS-14118 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Fengnan Li >Assignee: Fengnan Li >Priority: Major > Attachments: DNS testing log, HDFS design doc_ Single domain name for > clients - Google Docs-1.pdf, HDFS design doc_ Single domain name for clients > - Google Docs.pdf, HDFS-14118.001.patch, HDFS-14118.002.patch, > HDFS-14118.003.patch, HDFS-14118.004.patch, HDFS-14118.005.patch, > HDFS-14118.006.patch, HDFS-14118.007.patch, HDFS-14118.008.patch, > HDFS-14118.009.patch, HDFS-14118.010.patch, HDFS-14118.011.patch, > HDFS-14118.012.patch, HDFS-14118.013.patch, HDFS-14118.014.patch, > HDFS-14118.015.patch, HDFS-14118.016.patch, HDFS-14118.017.patch, > HDFS-14118.018.patch, HDFS-14118.019.patch, HDFS-14118.020.patch, > HDFS-14118.021.patch, HDFS-14118.patch > > > Clients will need to know about routers to talk to the HDFS cluster > (obviously), and having routers updating (adding/removing) will have to make > every client change, which is a painful process. > DNS can be used here to resolve the single domain name clients knows to a > list of routers in the current config. However, DNS won't be able to consider > only resolving to the working router based on certain health thresholds. > There are some ways about how this can be solved. One way is to have a > separate script to regularly check the status of the router and update the > DNS records if a router fails the health thresholds. In this way, security > might be carefully considered for this way. Another way is to have the client > do the normal connecting/failover after they get the list of routers, which > requires the change of current failover proxy provider. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14052) RBF: Use Router keytab for WebHDFS
[
https://issues.apache.org/jira/browse/HDFS-14052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16774866#comment-16774866
]
Hadoop QA commented on HDFS-14052:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-13891 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
16s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
30s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
33s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 51s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
53s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green} HDFS-13891 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 9s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 22m
20s{color} | {color:green} hadoop-hdfs-rbf in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 70m 15s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-14052 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12959721/HDFS-14052-HDFS-13891.2.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux ff1db07e1940 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | HDFS-13891 / 0477b0b |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/26302/testReport/ |
| Max. process+thread count | 1360 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U:
hadoop-hdfs-project/hadoop-hdfs-rbf |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/26302/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> RBF: Use Router keytab for WebHDFS
> --
>
> Key: HDFS-14052
>
[jira] [Commented] (HDFS-12943) Consistent Reads from Standby Node
[ https://issues.apache.org/jira/browse/HDFS-12943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16774857#comment-16774857 ] Zhe Zhang commented on HDFS-12943: -- [~vagarychen] has tested the current version of the feature on a real cluster, and can verify the aspects that have already been verified. I think [~jojochuang] has also done some tests. > Consistent Reads from Standby Node > -- > > Key: HDFS-12943 > URL: https://issues.apache.org/jira/browse/HDFS-12943 > Project: Hadoop HDFS > Issue Type: New Feature > Components: hdfs >Reporter: Konstantin Shvachko >Assignee: Konstantin Shvachko >Priority: Major > Attachments: ConsistentReadsFromStandbyNode.pdf, > ConsistentReadsFromStandbyNode.pdf, HDFS-12943-001.patch, > HDFS-12943-002.patch, HDFS-12943-003.patch, HDFS-12943-004.patch, > TestPlan-ConsistentReadsFromStandbyNode.pdf > > > StandbyNode in HDFS is a replica of the active NameNode. The states of the > NameNodes are coordinated via the journal. It is natural to consider > StandbyNode as a read-only replica. As with any replicated distributed system > the problem of stale reads should be resolved. Our main goal is to provide > reads from standby in a consistent way in order to enable a wide range of > existing applications running on top of HDFS. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14182) Datanode usage histogram is clicked to show ip list
[ https://issues.apache.org/jira/browse/HDFS-14182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16774851#comment-16774851 ] Yiqun Lin commented on HDFS-14182: -- [~fengchuang], I apply the change in my local. Only one consideration: if we click one histogram of dn which contains more than thousands of nodes, how will be ip hosts showed in the page? Can you take a look for this and attach the screenshot? > Datanode usage histogram is clicked to show ip list > --- > > Key: HDFS-14182 > URL: https://issues.apache.org/jira/browse/HDFS-14182 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: fengchuang >Assignee: fengchuang >Priority: Major > Attachments: HDFS-14182.001.patch, HDFS-14182.002.patch, > HDFS-14182.003.patch, showip.jpeg > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-14182) Datanode usage histogram is clicked to show ip list
[ https://issues.apache.org/jira/browse/HDFS-14182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16774851#comment-16774851 ] Yiqun Lin edited comment on HDFS-14182 at 2/22/19 7:15 AM: --- [~fengchuang], I apply the change in my local. It looks good when there are few nodes in cluster. Only one consideration: if we click one histogram of dn which contains more than thousands of nodes, how will be ip hosts showed in the page? Can you take a look for this and attach the screenshot? was (Author: linyiqun): [~fengchuang], I apply the change in my local. Only one consideration: if we click one histogram of dn which contains more than thousands of nodes, how will be ip hosts showed in the page? Can you take a look for this and attach the screenshot? > Datanode usage histogram is clicked to show ip list > --- > > Key: HDFS-14182 > URL: https://issues.apache.org/jira/browse/HDFS-14182 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: fengchuang >Assignee: fengchuang >Priority: Major > Attachments: HDFS-14182.001.patch, HDFS-14182.002.patch, > HDFS-14182.003.patch, showip.jpeg > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-14118) Use DNS to resolve Namenodes and Routers
[ https://issues.apache.org/jira/browse/HDFS-14118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16774848#comment-16774848 ] Yongjun Zhang edited comment on HDFS-14118 at 2/22/19 7:01 AM: --- Thanks for the explanation and new rev [~fengnanli]. A few more comments: # In hdfs-default.xml, remove the statement about "Random order should be ..." for both dfs.client.failover.resolve-needed and dfs.client.failover.resolver.impl, since this is already mentioned in the description of dfs.client.failover.random.order. # The description for dfs.client.failover.random.order can be improved to provide some more details about when to enable random and when not to. For example, for HA Namenodes, we may not need to enable; for ObserverNodes and RBF routers, it may be helpful to enable because we want to spread the load. This can be deferred to a new jira if you'd like. # Create a new Jira to fix the relevant doc places as [~elgoiri] pointed out [here | https://issues.apache.org/jira/browse/HDFS-14118?focusedCommentId=16773396=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16773396] . My +1 after that. Thanks. was (Author: yzhangal): Thanks for the explanation and new rev [~fengnanli]. A few more comments: # In hdfs-default.xml, remove the statement about "Random order should be ..." for both dfs.client.failover.resolve-needed and dfs.client.failover.resolver.impl, since this is already mentioned in the description of dfs.client.failover.random.order. # The description for dfs.client.failover.random.order can be improved to provide some more details about when to enable random and when not to. For example, for HA Namenodes, we may not need to enable; for ObserverNodes and RBF routers, it may be helpful to enable because we want to spread the load. This can be deferred to a new jira if you'd like. # Create a new Jira to fix the relevant doc places as [~elgoiri] pointed out here . My +1 after that. Thanks. > Use DNS to resolve Namenodes and Routers > > > Key: HDFS-14118 > URL: https://issues.apache.org/jira/browse/HDFS-14118 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Fengnan Li >Assignee: Fengnan Li >Priority: Major > Attachments: DNS testing log, HDFS design doc_ Single domain name for > clients - Google Docs-1.pdf, HDFS design doc_ Single domain name for clients > - Google Docs.pdf, HDFS-14118.001.patch, HDFS-14118.002.patch, > HDFS-14118.003.patch, HDFS-14118.004.patch, HDFS-14118.005.patch, > HDFS-14118.006.patch, HDFS-14118.007.patch, HDFS-14118.008.patch, > HDFS-14118.009.patch, HDFS-14118.010.patch, HDFS-14118.011.patch, > HDFS-14118.012.patch, HDFS-14118.013.patch, HDFS-14118.014.patch, > HDFS-14118.015.patch, HDFS-14118.016.patch, HDFS-14118.017.patch, > HDFS-14118.018.patch, HDFS-14118.019.patch, HDFS-14118.020.patch, > HDFS-14118.patch > > > Clients will need to know about routers to talk to the HDFS cluster > (obviously), and having routers updating (adding/removing) will have to make > every client change, which is a painful process. > DNS can be used here to resolve the single domain name clients knows to a > list of routers in the current config. However, DNS won't be able to consider > only resolving to the working router based on certain health thresholds. > There are some ways about how this can be solved. One way is to have a > separate script to regularly check the status of the router and update the > DNS records if a router fails the health thresholds. In this way, security > might be carefully considered for this way. Another way is to have the client > do the normal connecting/failover after they get the list of routers, which > requires the change of current failover proxy provider. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-14118) Use DNS to resolve Namenodes and Routers
[ https://issues.apache.org/jira/browse/HDFS-14118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16774848#comment-16774848 ] Yongjun Zhang edited comment on HDFS-14118 at 2/22/19 7:03 AM: --- Thanks for the explanation and new rev [~fengnanli]. A few more comments: # In hdfs-default.xml, remove the statement about "Random order should be ..." for both dfs.client.failover.resolve-needed and dfs.client.failover.resolver.impl, since this is already mentioned in the description of dfs.client.failover.random.order. # The description for dfs.client.failover.random.order can be improved to provide some more details about when to enable random and when not to. For example, for HA Namenodes, we may not need to enable; for ObserverNodes and RBF routers, it may be helpful to enable because we want to spread the load. This can be deferred to a new jira if you'd like. # Create a new Jira to fix the relevant doc places as [~elgoiri] pointed out here , plus relevant doc about read on standbyNN. My +1 after that. Thanks. was (Author: yzhangal): Thanks for the explanation and new rev [~fengnanli]. A few more comments: # In hdfs-default.xml, remove the statement about "Random order should be ..." for both dfs.client.failover.resolve-needed and dfs.client.failover.resolver.impl, since this is already mentioned in the description of dfs.client.failover.random.order. # The description for dfs.client.failover.random.order can be improved to provide some more details about when to enable random and when not to. For example, for HA Namenodes, we may not need to enable; for ObserverNodes and RBF routers, it may be helpful to enable because we want to spread the load. This can be deferred to a new jira if you'd like. # Create a new Jira to fix the relevant doc places as [~elgoiri] pointed out [here | https://issues.apache.org/jira/browse/HDFS-14118?focusedCommentId=16773396=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16773396] . My +1 after that. Thanks. > Use DNS to resolve Namenodes and Routers > > > Key: HDFS-14118 > URL: https://issues.apache.org/jira/browse/HDFS-14118 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Fengnan Li >Assignee: Fengnan Li >Priority: Major > Attachments: DNS testing log, HDFS design doc_ Single domain name for > clients - Google Docs-1.pdf, HDFS design doc_ Single domain name for clients > - Google Docs.pdf, HDFS-14118.001.patch, HDFS-14118.002.patch, > HDFS-14118.003.patch, HDFS-14118.004.patch, HDFS-14118.005.patch, > HDFS-14118.006.patch, HDFS-14118.007.patch, HDFS-14118.008.patch, > HDFS-14118.009.patch, HDFS-14118.010.patch, HDFS-14118.011.patch, > HDFS-14118.012.patch, HDFS-14118.013.patch, HDFS-14118.014.patch, > HDFS-14118.015.patch, HDFS-14118.016.patch, HDFS-14118.017.patch, > HDFS-14118.018.patch, HDFS-14118.019.patch, HDFS-14118.020.patch, > HDFS-14118.patch > > > Clients will need to know about routers to talk to the HDFS cluster > (obviously), and having routers updating (adding/removing) will have to make > every client change, which is a painful process. > DNS can be used here to resolve the single domain name clients knows to a > list of routers in the current config. However, DNS won't be able to consider > only resolving to the working router based on certain health thresholds. > There are some ways about how this can be solved. One way is to have a > separate script to regularly check the status of the router and update the > DNS records if a router fails the health thresholds. In this way, security > might be carefully considered for this way. Another way is to have the client > do the normal connecting/failover after they get the list of routers, which > requires the change of current failover proxy provider. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14118) Use DNS to resolve Namenodes and Routers
[ https://issues.apache.org/jira/browse/HDFS-14118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16774848#comment-16774848 ] Yongjun Zhang commented on HDFS-14118: -- Thanks for the explanation and new rev [~fengnanli]. A few more comments: # In hdfs-default.xml, remove the statement about "Random order should be ..." for both dfs.client.failover.resolve-needed and dfs.client.failover.resolver.impl, since this is already mentioned in the description of dfs.client.failover.random.order. # The description for dfs.client.failover.random.order can be improved to provide some more details about when to enable random and when not to. For example, for HA Namenodes, we may not need to enable; for ObserverNodes and RBF routers, it may be helpful to enable because we want to spread the load. This can be deferred to a new jira if you'd like. # Create a new Jira to fix the relevant doc places as [~elgoiri] pointed out [here | https://issues.apache.org/jira/browse/HDFS-14118?focusedCommentId=16773396=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16773396]. My +1 after that. Thanks. > Use DNS to resolve Namenodes and Routers > > > Key: HDFS-14118 > URL: https://issues.apache.org/jira/browse/HDFS-14118 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Fengnan Li >Assignee: Fengnan Li >Priority: Major > Attachments: DNS testing log, HDFS design doc_ Single domain name for > clients - Google Docs-1.pdf, HDFS design doc_ Single domain name for clients > - Google Docs.pdf, HDFS-14118.001.patch, HDFS-14118.002.patch, > HDFS-14118.003.patch, HDFS-14118.004.patch, HDFS-14118.005.patch, > HDFS-14118.006.patch, HDFS-14118.007.patch, HDFS-14118.008.patch, > HDFS-14118.009.patch, HDFS-14118.010.patch, HDFS-14118.011.patch, > HDFS-14118.012.patch, HDFS-14118.013.patch, HDFS-14118.014.patch, > HDFS-14118.015.patch, HDFS-14118.016.patch, HDFS-14118.017.patch, > HDFS-14118.018.patch, HDFS-14118.019.patch, HDFS-14118.020.patch, > HDFS-14118.patch > > > Clients will need to know about routers to talk to the HDFS cluster > (obviously), and having routers updating (adding/removing) will have to make > every client change, which is a painful process. > DNS can be used here to resolve the single domain name clients knows to a > list of routers in the current config. However, DNS won't be able to consider > only resolving to the working router based on certain health thresholds. > There are some ways about how this can be solved. One way is to have a > separate script to regularly check the status of the router and update the > DNS records if a router fails the health thresholds. In this way, security > might be carefully considered for this way. Another way is to have the client > do the normal connecting/failover after they get the list of routers, which > requires the change of current failover proxy provider. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-14118) Use DNS to resolve Namenodes and Routers
[ https://issues.apache.org/jira/browse/HDFS-14118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16774848#comment-16774848 ] Yongjun Zhang edited comment on HDFS-14118 at 2/22/19 6:58 AM: --- Thanks for the explanation and new rev [~fengnanli]. A few more comments: # In hdfs-default.xml, remove the statement about "Random order should be ..." for both dfs.client.failover.resolve-needed and dfs.client.failover.resolver.impl, since this is already mentioned in the description of dfs.client.failover.random.order. # The description for dfs.client.failover.random.order can be improved to provide some more details about when to enable random and when not to. For example, for HA Namenodes, we may not need to enable; for ObserverNodes and RBF routers, it may be helpful to enable because we want to spread the load. This can be deferred to a new jira if you'd like. # Create a new Jira to fix the relevant doc places as [~elgoiri] pointed out here . My +1 after that. Thanks. was (Author: yzhangal): Thanks for the explanation and new rev [~fengnanli]. A few more comments: # In hdfs-default.xml, remove the statement about "Random order should be ..." for both dfs.client.failover.resolve-needed and dfs.client.failover.resolver.impl, since this is already mentioned in the description of dfs.client.failover.random.order. # The description for dfs.client.failover.random.order can be improved to provide some more details about when to enable random and when not to. For example, for HA Namenodes, we may not need to enable; for ObserverNodes and RBF routers, it may be helpful to enable because we want to spread the load. This can be deferred to a new jira if you'd like. # Create a new Jira to fix the relevant doc places as [~elgoiri] pointed out [here | https://issues.apache.org/jira/browse/HDFS-14118?focusedCommentId=16773396=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16773396]. My +1 after that. Thanks. > Use DNS to resolve Namenodes and Routers > > > Key: HDFS-14118 > URL: https://issues.apache.org/jira/browse/HDFS-14118 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Fengnan Li >Assignee: Fengnan Li >Priority: Major > Attachments: DNS testing log, HDFS design doc_ Single domain name for > clients - Google Docs-1.pdf, HDFS design doc_ Single domain name for clients > - Google Docs.pdf, HDFS-14118.001.patch, HDFS-14118.002.patch, > HDFS-14118.003.patch, HDFS-14118.004.patch, HDFS-14118.005.patch, > HDFS-14118.006.patch, HDFS-14118.007.patch, HDFS-14118.008.patch, > HDFS-14118.009.patch, HDFS-14118.010.patch, HDFS-14118.011.patch, > HDFS-14118.012.patch, HDFS-14118.013.patch, HDFS-14118.014.patch, > HDFS-14118.015.patch, HDFS-14118.016.patch, HDFS-14118.017.patch, > HDFS-14118.018.patch, HDFS-14118.019.patch, HDFS-14118.020.patch, > HDFS-14118.patch > > > Clients will need to know about routers to talk to the HDFS cluster > (obviously), and having routers updating (adding/removing) will have to make > every client change, which is a painful process. > DNS can be used here to resolve the single domain name clients knows to a > list of routers in the current config. However, DNS won't be able to consider > only resolving to the working router based on certain health thresholds. > There are some ways about how this can be solved. One way is to have a > separate script to regularly check the status of the router and update the > DNS records if a router fails the health thresholds. In this way, security > might be carefully considered for this way. Another way is to have the client > do the normal connecting/failover after they get the list of routers, which > requires the change of current failover proxy provider. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1163) Basic framework for Ozone Data Scrubber
[ https://issues.apache.org/jira/browse/HDDS-1163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Supratim Deka updated HDDS-1163: Description: Included in the scope: 1. Background scanner thread to iterate over container set and dispatch check tasks for individual containers 2. Fixed rate scheduling - dispatch tasks at a pre-determined rate (for example 1 container/s) 3. Check disk layout of Container - basic check for integrity of the directory hierarchy inside the container, include chunk directory and metadata directories 4. Check container file - basic sanity checks for the container metafile 5. Check Block Database - iterate over entries in the container block database and check for the existence and accessibility of the chunks for each block. Not in scope (will be done as separate subtasks): 1. Dynamic scheduling/pacing of background scan based on system load and available resources. 2. Detection and handling of orphan chunks 3. Checksum verification for Chunks 4. Corruption handling - reporting (to SCM) and subsequent handling of any corruption detected by the scanner. The current subtask will simply log any corruption which is detected. was: Included in the scope: 1. Background scanner thread to iterate over container set and dispatch check tasks for individual containers 2. Fixed rate scheduling - dispatch tasks at a pre-determined rate (for example 1 container/s) 3. Check disk layout of Container - basic check for integrity of the directory hierarchy inside the container, include chunk directory and metadata directories 4. Check container file - basic sanity checks for the container metafile 5. Check Block Database - iterate over entries in the container block database and check for the existence and accessibility of the chunks for each block Not in scope (will be done as separate subtasks): 1. Dynamic scheduling/pacing of background scan based on system load and available resources. 2. Detection and handling of orphan chunks 3. Checksum verification for Chunks > Basic framework for Ozone Data Scrubber > --- > > Key: HDDS-1163 > URL: https://issues.apache.org/jira/browse/HDDS-1163 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Datanode >Reporter: Supratim Deka >Assignee: Supratim Deka >Priority: Major > > Included in the scope: > 1. Background scanner thread to iterate over container set and dispatch check > tasks for individual containers > 2. Fixed rate scheduling - dispatch tasks at a pre-determined rate (for > example 1 container/s) > 3. Check disk layout of Container - basic check for integrity of the > directory hierarchy inside the container, include chunk directory and > metadata directories > 4. Check container file - basic sanity checks for the container metafile > 5. Check Block Database - iterate over entries in the container block > database and check for the existence and accessibility of the chunks for each > block. > Not in scope (will be done as separate subtasks): > 1. Dynamic scheduling/pacing of background scan based on system load and > available resources. > 2. Detection and handling of orphan chunks > 3. Checksum verification for Chunks > 4. Corruption handling - reporting (to SCM) and subsequent handling of any > corruption detected by the scanner. The current subtask will simply log any > corruption which is detected. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-1163) Basic framework for Ozone Data Scrubber
Supratim Deka created HDDS-1163: --- Summary: Basic framework for Ozone Data Scrubber Key: HDDS-1163 URL: https://issues.apache.org/jira/browse/HDDS-1163 Project: Hadoop Distributed Data Store Issue Type: Sub-task Components: Ozone Datanode Reporter: Supratim Deka Assignee: Supratim Deka Included in the scope: 1. Background scanner thread to iterate over container set and dispatch check tasks for individual containers 2. Fixed rate scheduling - dispatch tasks at a pre-determined rate (for example 1 container/s) 3. Check disk layout of Container - basic check for integrity of the directory hierarchy inside the container, include chunk directory and metadata directories 4. Check container file - basic sanity checks for the container metafile 5. Check Block Database - iterate over entries in the container block database and check for the existence and accessibility of the chunks for each block Not in scope (will be done as separate subtasks): 1. Dynamic scheduling/pacing of background scan based on system load and available resources. 2. Detection and handling of orphan chunks 3. Checksum verification for Chunks -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14305) Serial number in BlockTokenSecretManager could overlap between different namenodes
[
https://issues.apache.org/jira/browse/HDFS-14305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16774839#comment-16774839
]
Chao Sun commented on HDFS-14305:
-
[~jojochuang] this problem is not related to observer nodes - it could happen
to any HA cluster with the multi-SBN feature. The chance for the collision to
happen is very low though.
> Serial number in BlockTokenSecretManager could overlap between different
> namenodes
> --
>
> Key: HDFS-14305
> URL: https://issues.apache.org/jira/browse/HDFS-14305
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: security
>Reporter: Chao Sun
>Assignee: Chao Sun
>Priority: Major
>
> Currently, a {{BlockTokenSecretManager}} starts with a random integer as the
> initial serial number, and then use this formula to rotate it:
> {code:java}
> this.intRange = Integer.MAX_VALUE / numNNs;
> this.nnRangeStart = intRange * nnIndex;
> this.serialNo = (this.serialNo % intRange) + (nnRangeStart);
> {code}
> while {{numNNs}} is the total number of NameNodes in the cluster, and
> {{nnIndex}} is the index of the current NameNode specified in the
> configuration {{dfs.ha.namenodes.}}.
> However, with this approach, different NameNode could have overlapping ranges
> for serial number. For simplicity, let's assume {{Integer.MAX_VALUE}} is 100,
> and we have 2 NameNodes {{nn1}} and {{nn2}} in configuration. Then the ranges
> for these two are:
> {code}
> nn1 -> [-49, 49]
> nn2 -> [1, 99]
> {code}
> This is because the initial serial number could be any negative integer.
> Moreover, when the keys are updated, the serial number will again be updated
> with the formula:
> {code}
> this.serialNo = (this.serialNo % intRange) + (nnRangeStart);
> {code}
> which means the new serial number could be updated to a range that belongs to
> a different NameNode, thus increasing the chance of collision again.
> When the collision happens, DataNodes could overwrite an existing key which
> will cause clients to fail because of {{InvalidToken}} error.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-1162) Data Scrubbing for Ozone Containers
Supratim Deka created HDDS-1162: --- Summary: Data Scrubbing for Ozone Containers Key: HDDS-1162 URL: https://issues.apache.org/jira/browse/HDDS-1162 Project: Hadoop Distributed Data Store Issue Type: New Feature Components: Ozone Datanode, SCM Reporter: Supratim Deka Attachments: Container Scanner Redux.docx General Reference: https://en.wikipedia.org/wiki/Data_scrubbing The internal Design Document for Ozone Container Scanning is attached. Also link below: https://docs.google.com/document/d/1LLGCuGq7n3TVlJcxyUR1LgUhwqC5WB_OwRqQkRw3Eng/edit# -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14309) name node fail over failed with ssh fence failed because of jsch login failed with key check
[ https://issues.apache.org/jira/browse/HDFS-14309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16774825#comment-16774825 ] He Xiaoqiao commented on HDFS-14309: Hi [~iamgd67], Thanks to report this issue. HADOOP-14100 has fixed it, and also merge into branch-2.7, I think you could backport HADOOP-14100 to your own branch. FYI. > name node fail over failed with ssh fence failed because of jsch login failed > with key check > > > Key: HDFS-14309 > URL: https://issues.apache.org/jira/browse/HDFS-14309 > Project: Hadoop HDFS > Issue Type: Bug > Components: auto-failover >Affects Versions: 2.7.3 > Environment: linux CentOS release 6.8 (Final) kernel > 2.6.32-642.6.1.el6.x86_64 > >Reporter: qiang Liu >Priority: Major > Attachments: HDFS-14309-branch-2.7.3.001.patch > > Original Estimate: 1m > Remaining Estimate: 1m > > name node fail over failed with ssh fence failed because of jsch login failed > with key check. > the loged error is "Algorithm negotiation fail" > update jsch to 0.1.54 works ok -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14052) RBF: Use Router keytab for WebHDFS
[
https://issues.apache.org/jira/browse/HDFS-14052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
CR Hota updated HDFS-14052:
---
Attachment: HDFS-14052-HDFS-13891.2.patch
> RBF: Use Router keytab for WebHDFS
> --
>
> Key: HDFS-14052
> URL: https://issues.apache.org/jira/browse/HDFS-14052
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: CR Hota
>Priority: Major
> Attachments: HDFS-14052-HDFS-13891.0.patch,
> HDFS-14052-HDFS-13891.1.patch, HDFS-14052-HDFS-13891.2.patch
>
>
> When the RouterHttpServer starts it does:
> {code}
> NameNodeHttpServer.initWebHdfs(conf, httpAddress.getHostName(),
> httpServer,
> RouterWebHdfsMethods.class.getPackage().getName());
> {code}
> This function is in the NN and is pretty generic.
> However, it then calls to NameNodeHttpServer#getAuthFilterParams, which does:
> {code}
> String httpKeytab = conf.get(DFSUtil.getSpnegoKeytabKey(conf,
> DFSConfigKeys.DFS_NAMENODE_KEYTAB_FILE_KEY));
> {code}
> In most cases, the regular web keytab will kick in, but we should make this a
> parameter and load the Router one just in case.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14201) Ability to disallow safemode NN to become active
[ https://issues.apache.org/jira/browse/HDFS-14201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16774799#comment-16774799 ] He Xiaoqiao commented on HDFS-14201: Hi [~surmountian], [^HDFS-14201.001.patch] is my improvement for online production env, and I have turn off ability about #transitionToActive when namenode is still in safemode. It looks work well for over half a years. FYI. > Ability to disallow safemode NN to become active > > > Key: HDFS-14201 > URL: https://issues.apache.org/jira/browse/HDFS-14201 > Project: Hadoop HDFS > Issue Type: Improvement > Components: auto-failover >Affects Versions: 3.1.1, 2.9.2 >Reporter: Xiao Liang >Assignee: Xiao Liang >Priority: Major > Attachments: HDFS-14201.001.patch > > > Currently with HA, Namenode in safemode can be possibly selected as active, > for availability of both read and write, Namenodes not in safemode are better > choices to become active though. > It can take tens of minutes for a cold started Namenode to get out of > safemode, especially when there are large number of files and blocks in HDFS, > that means if a Namenode in safemode become active, the cluster will be not > fully functioning for quite a while, even if it can while there is some > Namenode not in safemode. > The proposal here is to add an option, to allow Namenode to report itself as > UNHEALTHY to ZKFC, if it's in safemode, so as to only allow fully functioning > Namenode to become active, improving the general availability of the cluster. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14201) Ability to disallow safemode NN to become active
[ https://issues.apache.org/jira/browse/HDFS-14201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] He Xiaoqiao updated HDFS-14201: --- Attachment: HDFS-14201.001.patch > Ability to disallow safemode NN to become active > > > Key: HDFS-14201 > URL: https://issues.apache.org/jira/browse/HDFS-14201 > Project: Hadoop HDFS > Issue Type: Improvement > Components: auto-failover >Affects Versions: 3.1.1, 2.9.2 >Reporter: Xiao Liang >Assignee: Xiao Liang >Priority: Major > Attachments: HDFS-14201.001.patch > > > Currently with HA, Namenode in safemode can be possibly selected as active, > for availability of both read and write, Namenodes not in safemode are better > choices to become active though. > It can take tens of minutes for a cold started Namenode to get out of > safemode, especially when there are large number of files and blocks in HDFS, > that means if a Namenode in safemode become active, the cluster will be not > fully functioning for quite a while, even if it can while there is some > Namenode not in safemode. > The proposal here is to add an option, to allow Namenode to report itself as > UNHEALTHY to ZKFC, if it's in safemode, so as to only allow fully functioning > Namenode to become active, improving the general availability of the cluster. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14295) Add Threadpool for DataTransfers
[
https://issues.apache.org/jira/browse/HDFS-14295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16774784#comment-16774784
]
Hadoop QA commented on HDFS-14295:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
37s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 47s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 27s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 76m 28s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
35s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}129m 47s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.qjournal.server.TestJournalNodeSync |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-14295 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12959706/HDFS-14295.5.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux f3683ee8e79a 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 28d0bf9 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/26300/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/26300/testReport/ |
| Max. process+thread count | 4529 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
[jira] [Work logged] (HDDS-919) Enable prometheus endpoints for Ozone datanodes
[ https://issues.apache.org/jira/browse/HDDS-919?focusedWorklogId=202376=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-202376 ] ASF GitHub Bot logged work on HDDS-919: --- Author: ASF GitHub Bot Created on: 22/Feb/19 04:33 Start Date: 22/Feb/19 04:33 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on pull request #502: HDDS-919. Enable prometheus endpoints for Ozone datanodes URL: https://github.com/apache/hadoop/pull/502#discussion_r259209954 ## File path: hadoop-hdds/common/src/main/resources/ozone-default.xml ## @@ -1881,4 +1881,71 @@ jar and false for the ozone-filesystem-lib.jar + + + +hdds.datanode.http.kerberos.principal +HTTP/_h...@example.com + Review comment: Can we add tag as Ozone,Security? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 202376) Time Spent: 50m (was: 40m) > Enable prometheus endpoints for Ozone datanodes > --- > > Key: HDDS-919 > URL: https://issues.apache.org/jira/browse/HDDS-919 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Labels: newbie, pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > HDDS-846 provides a new metric endpoint which publishes the available Hadoop > metrics in prometheus friendly format with a new servlet. > Unfortunately it's enabled only on the scm/om side. It would be great to > enable it in the Ozone/HDDS datanodes on the web server of the HDDS Rest > endpoint. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14292) Introduce Java ExecutorService to DataXceiverServer
[
https://issues.apache.org/jira/browse/HDFS-14292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16774761#comment-16774761
]
Hadoop QA commented on HDFS-14292:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 0s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
20s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
54s{color} | {color:green} hadoop-hdfs-project generated 0 new + 537 unchanged
- 3 fixed = 537 total (was 540) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 9s{color} | {color:orange} hadoop-hdfs-project: The patch generated 3 new +
630 unchanged - 8 fixed = 633 total (was 638) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 34s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
52s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 75m 53s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
36s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}143m 12s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.server.namenode.TestNamenodeCapacityReport |
| | hadoop.hdfs.qjournal.server.TestJournalNodeSync |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-14292 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12959699/HDFS-14292.7.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle xml |
| uname | Linux
[jira] [Work logged] (HDDS-919) Enable prometheus endpoints for Ozone datanodes
[ https://issues.apache.org/jira/browse/HDDS-919?focusedWorklogId=202378=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-202378 ] ASF GitHub Bot logged work on HDDS-919: --- Author: ASF GitHub Bot Created on: 22/Feb/19 04:34 Start Date: 22/Feb/19 04:34 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on pull request #502: HDDS-919. Enable prometheus endpoints for Ozone datanodes URL: https://github.com/apache/hadoop/pull/502#discussion_r259210266 ## File path: hadoop-hdds/common/src/main/resources/ozone-default.xml ## @@ -1881,4 +1881,71 @@ jar and false for the ozone-filesystem-lib.jar + + + +hdds.datanode.http.kerberos.principal +HTTP/_h...@example.com + + The kerberos principal for the datanode http server. + + + + +hdds.datanode.http.kerberos.keytab +/etc/security/keytabs/HTTP.keytab + + The kerberos keytab file for datanode http server + + + + +hdds.datanode.http-address +0.0.0.0:9882 +OZONE, MANAGEMENT + + The address and the base port where the Datanode web ui will listen on. + + If the port is 0 then the server will start on a free port. + + + +hdds.datanode.http-bind-host +0.0.0.0 +OZONE, MANAGEMENT + + The actual address the Datanode web server will bind to. If this + optional address is set, it overrides only the hostname portion of + hdds.datanode.http-address. + + + +hdds.datanode.http.enabled +true +OZONE, MANAGEMENT + + Property to enable or disable Datanode web ui. + + + +hdds.datanode.https-address +0.0.0.0:9883 +OZONE, MANAGEMENT + + The address and the base port where the Datanode web UI will listen + on using HTTPS. + + If the port is 0 then the server will start on a free port. + + + +hdds.datanode.https-bind-host +0.0.0.0 +OZONE, MANAGEMENT Review comment: for https keys, can we add SECURITY tag also? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 202378) Time Spent: 1h 10m (was: 1h) > Enable prometheus endpoints for Ozone datanodes > --- > > Key: HDDS-919 > URL: https://issues.apache.org/jira/browse/HDDS-919 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Labels: newbie, pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > HDDS-846 provides a new metric endpoint which publishes the available Hadoop > metrics in prometheus friendly format with a new servlet. > Unfortunately it's enabled only on the scm/om side. It would be great to > enable it in the Ozone/HDDS datanodes on the web server of the HDDS Rest > endpoint. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-919) Enable prometheus endpoints for Ozone datanodes
[ https://issues.apache.org/jira/browse/HDDS-919?focusedWorklogId=202375=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-202375 ] ASF GitHub Bot logged work on HDDS-919: --- Author: ASF GitHub Bot Created on: 22/Feb/19 04:33 Start Date: 22/Feb/19 04:33 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on pull request #502: HDDS-919. Enable prometheus endpoints for Ozone datanodes URL: https://github.com/apache/hadoop/pull/502#discussion_r259209970 ## File path: hadoop-hdds/common/src/main/resources/ozone-default.xml ## @@ -1881,4 +1881,71 @@ jar and false for the ozone-filesystem-lib.jar + + + +hdds.datanode.http.kerberos.principal +HTTP/_h...@example.com + + The kerberos principal for the datanode http server. + + + + +hdds.datanode.http.kerberos.keytab +/etc/security/keytabs/HTTP.keytab Review comment: Can we add tag as Ozone,Security? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 202375) Time Spent: 40m (was: 0.5h) > Enable prometheus endpoints for Ozone datanodes > --- > > Key: HDDS-919 > URL: https://issues.apache.org/jira/browse/HDDS-919 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Labels: newbie, pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > HDDS-846 provides a new metric endpoint which publishes the available Hadoop > metrics in prometheus friendly format with a new servlet. > Unfortunately it's enabled only on the scm/om side. It would be great to > enable it in the Ozone/HDDS datanodes on the web server of the HDDS Rest > endpoint. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-919) Enable prometheus endpoints for Ozone datanodes
[ https://issues.apache.org/jira/browse/HDDS-919?focusedWorklogId=202377=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-202377 ] ASF GitHub Bot logged work on HDDS-919: --- Author: ASF GitHub Bot Created on: 22/Feb/19 04:33 Start Date: 22/Feb/19 04:33 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on pull request #502: HDDS-919. Enable prometheus endpoints for Ozone datanodes URL: https://github.com/apache/hadoop/pull/502#discussion_r259210122 ## File path: hadoop-hdds/container-service/pom.xml ## @@ -37,6 +37,10 @@ http://maven.apache.org/xsd/maven-4.0.0.xsd;> org.apache.hadoop hadoop-hdds-server-framework + + org.apache.hadoop Review comment: I think it is duplicated, as above lines 37-38 already have added this dependency. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 202377) Time Spent: 1h (was: 50m) > Enable prometheus endpoints for Ozone datanodes > --- > > Key: HDDS-919 > URL: https://issues.apache.org/jira/browse/HDDS-919 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Labels: newbie, pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > HDDS-846 provides a new metric endpoint which publishes the available Hadoop > metrics in prometheus friendly format with a new servlet. > Unfortunately it's enabled only on the scm/om side. It would be great to > enable it in the Ozone/HDDS datanodes on the web server of the HDDS Rest > endpoint. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1152) Add trace information for the client side of the datanode writes
[ https://issues.apache.org/jira/browse/HDDS-1152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16774747#comment-16774747 ] Shashikant Banerjee commented on HDDS-1152: --- +1. Looks good to me. > Add trace information for the client side of the datanode writes > > > Key: HDDS-1152 > URL: https://issues.apache.org/jira/browse/HDDS-1152 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > The XCeiverClients can be traced on the client side to get some information > about the time of chunk writes / block puts. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14130) Make ZKFC ObserverNode aware
[
https://issues.apache.org/jira/browse/HDFS-14130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16774740#comment-16774740
]
Hadoop QA commented on HDFS-14130:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
5s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 20m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
19m 36s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
54s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 6s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
55s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
38s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 84m 50s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
43s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}206m 37s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS |
| | hadoop.hdfs.qjournal.server.TestJournalNodeSync |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-14130 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12959688/HDFS-14130.010.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux c0243454aba1 4.4.0-138-generic #164~14.04.1-Ubuntu SMP Fri Oct
5 08:56:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 28d0bf9 |
| maven | version: Apache Maven 3.3.9 |
| Default Java |
[jira] [Comment Edited] (HDDS-1038) Support Service Level Authorization for OM, SCM and DN
[
https://issues.apache.org/jira/browse/HDDS-1038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16774729#comment-16774729
]
Ajay Kumar edited comment on HDDS-1038 at 2/22/19 3:42 AM:
---
+1 pending jenkins
was (Author: ajayydv):
+1
> Support Service Level Authorization for OM, SCM and DN
> --
>
> Key: HDDS-1038
> URL: https://issues.apache.org/jira/browse/HDDS-1038
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Ajay Kumar
>Assignee: Ajay Kumar
>Priority: Major
> Labels: Security
> Fix For: 0.4.0
>
> Attachments: HDDS-1038.00.patch, HDDS-1038.01.patch,
> HDDS-1038.02.patch, HDDS-1038.03.patch, HDDS-1038.04.patch,
> HDDS-1038.05.patch, HDDS-1038.06.patch, HDDS-1038.07.patch
>
>
> In a secure Ozone cluster. Datanodes fail to connect to SCM on
> {{StorageContainerDatanodeProtocol}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1038) Support Service Level Authorization for OM, SCM and DN
[
https://issues.apache.org/jira/browse/HDDS-1038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16774729#comment-16774729
]
Ajay Kumar commented on HDDS-1038:
--
+1
> Support Service Level Authorization for OM, SCM and DN
> --
>
> Key: HDDS-1038
> URL: https://issues.apache.org/jira/browse/HDDS-1038
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Ajay Kumar
>Assignee: Ajay Kumar
>Priority: Major
> Labels: Security
> Fix For: 0.4.0
>
> Attachments: HDDS-1038.00.patch, HDDS-1038.01.patch,
> HDDS-1038.02.patch, HDDS-1038.03.patch, HDDS-1038.04.patch,
> HDDS-1038.05.patch, HDDS-1038.06.patch, HDDS-1038.07.patch
>
>
> In a secure Ozone cluster. Datanodes fail to connect to SCM on
> {{StorageContainerDatanodeProtocol}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14295) Add Threadpool for DataTransfers
[
https://issues.apache.org/jira/browse/HDFS-14295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-14295:
---
Status: Patch Available (was: Open)
[~elgoiri] I think the code is simple enough to pass with code-review alone. I
also think that unit tests shouldn't be too concerned with implementation
details like caching.
> Add Threadpool for DataTransfers
>
>
> Key: HDFS-14295
> URL: https://issues.apache.org/jira/browse/HDFS-14295
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14295.1.patch, HDFS-14295.2.patch,
> HDFS-14295.3.patch, HDFS-14295.4.patch, HDFS-14295.5.patch, HDFS-14295.5.patch
>
>
> When a DataNode data transfers a block, is spins up a new thread for each
> transfer.
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339]
> and
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022].
> Instead, add the threads to a {{CachedThreadPool}} so that when their
> threads complete the transfer, they can be re-used for another transfer. This
> should save resources spent on creating and spinning up transfer threads.
> One thing I'll point out that's a bit off, which I address in this patch, ...
> There are two places in the code where a {{DataTransfer}} thread is started.
> In [one
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339-L2341],
> it's started in a default thread group. In [another
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022],
> it's started in the
> [dataXceiverServer|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L1164]
> thread group.
> I do not think it's correct to include any of these threads in the
> {{dataXceiverServer}} thread group. Anything submitted to the
> {{dataXceiverServer}} should probably be tied to the
> {{dfs.datanode.max.transfer.threads}} configurations, and neither of these
> methods are. Instead, they should be submitted into the same thread pool with
> its own thread group (probably the default thread group, unless someone
> suggests otherwise) and is what I have included in this patch.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1070) Adding Node and Pipeline related metrics in SCM
[ https://issues.apache.org/jira/browse/HDDS-1070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16774719#comment-16774719 ] Yiqun Lin commented on HDDS-1070: - LGTM, +1 pending Jenkins. > Adding Node and Pipeline related metrics in SCM > --- > > Key: HDDS-1070 > URL: https://issues.apache.org/jira/browse/HDDS-1070 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: SCM >Affects Versions: 0.3.0 >Reporter: Nanda kumar >Assignee: Nanda kumar >Priority: Major > Attachments: HDDS-1070.000.patch, HDDS-1070.001.patch, > HDDS-1070.002.patch > > > This jira aims to add more Node and Pipeline related metrics to SCM. > Following metrics will be added as part of this jira: > * numberOfSuccessfulPipelineCreation > * numberOfFailedPipelineCreation > * numberOfSuccessfulPipelineDestroy > * numberOfFailedPipelineDestroy > * numberOfPipelineReportProcessed > * numberOfNodeReportProcessed > * numberOfHBProcessed > * number of pipelines in different PipelineState -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14292) Introduce Java ExecutorService to DataXceiverServer
[
https://issues.apache.org/jira/browse/HDFS-14292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16774692#comment-16774692
]
BELUGA BEHR commented on HDFS-14292:
Pushed a new patch to fix some of the unit tests.
Please note that this patch also includes changes to the {{Peer}} interface to
allow for some cleaner code.
As part of that, the logging has changed a bit and I think it is more clear.
It certainly is closer to a spec that most people would recognize.
{code}
from peer: NioInetPeer [isLocal=true, localURI=hdfs+dn://127.0.0.1:35375,
remoteURI=hdfs+dn://127.0.0.1:38376]
from peer: BasicInetPeer [isLocal=true, localURI=hdfs+dn://127.0.0.1:35375,
remoteURI=hdfs+dn://127.0.0.1:38376]
from peer: DomainPeer [isLocal=true,
localURI=hdfs+dn+unix://127.0.0.1/tmp/socket,
remoteURI=hdfs+dn+unix://127.0.0.1/tmp/socket]
{code}
These are stored in actual {{URI}} objects.
The {{hdfs+dn}} scheme is typical datagram socket stuff (host and port). The
{{hdfs+dn+unix}} specifies that the DataNode is communicating over a Unix
domain socket (file). There is no port obviously, but the URI path is the path
to the socket file.
> Introduce Java ExecutorService to DataXceiverServer
> ---
>
> Key: HDFS-14292
> URL: https://issues.apache.org/jira/browse/HDFS-14292
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14292.1.patch, HDFS-14292.2.patch,
> HDFS-14292.3.patch, HDFS-14292.4.patch, HDFS-14292.5.patch,
> HDFS-14292.6.patch, HDFS-14292.6.patch, HDFS-14292.7.patch
>
>
> I wanted to investigate {{dfs.datanode.max.transfer.threads}} from
> {{hdfs-site.xml}}. It is described as "Specifies the maximum number of
> threads to use for transferring data in and out of the DN." The default
> value is 4096. I found it interesting because 4096 threads sounds like a lot
> to me. I'm not sure how a system with 8-16 cores would react to this large a
> thread count. Intuitively, I would say that the overhead of context
> switching would be immense.
> During mt investigation, I discovered the
> [following|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java#L203-L216]
> setup in the {{DataXceiverServer}} class:
> # A peer connects to a DataNode
> # A new thread is spun up to service this connection
> # The thread runs to completion
> # The tread dies
> It would perhaps be better if we used a thread pool to better manage the
> lifecycle of the service threads and to allow the DataNode to re-use existing
> threads, saving on the need to create and spin-up threads on demand.
> In this JIRA, I have added a couple of things:
> # Added a thread pool to {{DataXceiverServer}} class that, on demand, will
> create up to {{dfs.datanode.max.transfer.threads}}. A thread that has
> completed its prior duties will stay idle for up to 60 seconds
> (configurable), it will be retired if no new work has arrived.
> # Added new methods to the {{Peer}} Interface to allow for better logging and
> less code within each Thread ({{DataXceiver}}).
> # Updated the Thread code ({{DataXceiver}}) regarding its interactions with
> {{blockReceiver}} instance variable
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14295) Add Threadpool for DataTransfers
[
https://issues.apache.org/jira/browse/HDFS-14295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-14295:
---
Attachment: HDFS-14295.5.patch
> Add Threadpool for DataTransfers
>
>
> Key: HDFS-14295
> URL: https://issues.apache.org/jira/browse/HDFS-14295
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14295.1.patch, HDFS-14295.2.patch,
> HDFS-14295.3.patch, HDFS-14295.4.patch, HDFS-14295.5.patch, HDFS-14295.5.patch
>
>
> When a DataNode data transfers a block, is spins up a new thread for each
> transfer.
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339]
> and
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022].
> Instead, add the threads to a {{CachedThreadPool}} so that when their
> threads complete the transfer, they can be re-used for another transfer. This
> should save resources spent on creating and spinning up transfer threads.
> One thing I'll point out that's a bit off, which I address in this patch, ...
> There are two places in the code where a {{DataTransfer}} thread is started.
> In [one
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339-L2341],
> it's started in a default thread group. In [another
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022],
> it's started in the
> [dataXceiverServer|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L1164]
> thread group.
> I do not think it's correct to include any of these threads in the
> {{dataXceiverServer}} thread group. Anything submitted to the
> {{dataXceiverServer}} should probably be tied to the
> {{dfs.datanode.max.transfer.threads}} configurations, and neither of these
> methods are. Instead, they should be submitted into the same thread pool with
> its own thread group (probably the default thread group, unless someone
> suggests otherwise) and is what I have included in this patch.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14295) Add Threadpool for DataTransfers
[
https://issues.apache.org/jira/browse/HDFS-14295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-14295:
---
Status: Open (was: Patch Available)
> Add Threadpool for DataTransfers
>
>
> Key: HDFS-14295
> URL: https://issues.apache.org/jira/browse/HDFS-14295
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14295.1.patch, HDFS-14295.2.patch,
> HDFS-14295.3.patch, HDFS-14295.4.patch, HDFS-14295.5.patch, HDFS-14295.5.patch
>
>
> When a DataNode data transfers a block, is spins up a new thread for each
> transfer.
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339]
> and
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022].
> Instead, add the threads to a {{CachedThreadPool}} so that when their
> threads complete the transfer, they can be re-used for another transfer. This
> should save resources spent on creating and spinning up transfer threads.
> One thing I'll point out that's a bit off, which I address in this patch, ...
> There are two places in the code where a {{DataTransfer}} thread is started.
> In [one
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339-L2341],
> it's started in a default thread group. In [another
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022],
> it's started in the
> [dataXceiverServer|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L1164]
> thread group.
> I do not think it's correct to include any of these threads in the
> {{dataXceiverServer}} thread group. Anything submitted to the
> {{dataXceiverServer}} should probably be tied to the
> {{dfs.datanode.max.transfer.threads}} configurations, and neither of these
> methods are. Instead, they should be submitted into the same thread pool with
> its own thread group (probably the default thread group, unless someone
> suggests otherwise) and is what I have included in this patch.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14295) Add Threadpool for DataTransfers
[
https://issues.apache.org/jira/browse/HDFS-14295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16774705#comment-16774705
]
Hadoop QA commented on HDFS-14295:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
33s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 5s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 22s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 80m 45s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
36s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}135m 22s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.qjournal.server.TestJournalNodeSync |
| | hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-14295 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12959691/HDFS-14295.5.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 99e9ca808a92 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 28d0bf9 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/26297/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/26297/testReport/ |
[jira] [Comment Edited] (HDFS-13123) RBF: Add a balancer tool to move data across subcluster
[ https://issues.apache.org/jira/browse/HDFS-13123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16774698#comment-16774698 ] Yiqun Lin edited comment on HDFS-13123 at 2/22/19 2:43 AM: --- Hi [~ywskycn] and [~elgoiri], is there any POC patch for this as I remember rebalancer has been used in your production env? I will follow-up this and do something if I can help, :). was (Author: linyiqun): Hi [~ywskycn] and [~elgoiri], is there any POC patch for this as I remember rebalancer has been used in your production env? > RBF: Add a balancer tool to move data across subcluster > > > Key: HDFS-13123 > URL: https://issues.apache.org/jira/browse/HDFS-13123 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Wei Yan >Assignee: Wei Yan >Priority: Major > Attachments: HDFS Router-Based Federation Rebalancer.pdf > > > Follow the discussion in HDFS-12615. This Jira is to track effort for > building a rebalancer tool, used by router-based federation to move data > among subclusters. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13123) RBF: Add a balancer tool to move data across subcluster
[ https://issues.apache.org/jira/browse/HDFS-13123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16774698#comment-16774698 ] Yiqun Lin commented on HDFS-13123: -- Hi [~ywskycn] and [~elgoiri], is there any POC patch for this as I remember rebalancer has been used in your production env? > RBF: Add a balancer tool to move data across subcluster > > > Key: HDFS-13123 > URL: https://issues.apache.org/jira/browse/HDFS-13123 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Wei Yan >Assignee: Wei Yan >Priority: Major > Attachments: HDFS Router-Based Federation Rebalancer.pdf > > > Follow the discussion in HDFS-12615. This Jira is to track effort for > building a rebalancer tool, used by router-based federation to move data > among subclusters. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14292) Introduce Java ExecutorService to DataXceiverServer
[
https://issues.apache.org/jira/browse/HDFS-14292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-14292:
---
Status: Open (was: Patch Available)
> Introduce Java ExecutorService to DataXceiverServer
> ---
>
> Key: HDFS-14292
> URL: https://issues.apache.org/jira/browse/HDFS-14292
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14292.1.patch, HDFS-14292.2.patch,
> HDFS-14292.3.patch, HDFS-14292.4.patch, HDFS-14292.5.patch,
> HDFS-14292.6.patch, HDFS-14292.6.patch, HDFS-14292.7.patch
>
>
> I wanted to investigate {{dfs.datanode.max.transfer.threads}} from
> {{hdfs-site.xml}}. It is described as "Specifies the maximum number of
> threads to use for transferring data in and out of the DN." The default
> value is 4096. I found it interesting because 4096 threads sounds like a lot
> to me. I'm not sure how a system with 8-16 cores would react to this large a
> thread count. Intuitively, I would say that the overhead of context
> switching would be immense.
> During mt investigation, I discovered the
> [following|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java#L203-L216]
> setup in the {{DataXceiverServer}} class:
> # A peer connects to a DataNode
> # A new thread is spun up to service this connection
> # The thread runs to completion
> # The tread dies
> It would perhaps be better if we used a thread pool to better manage the
> lifecycle of the service threads and to allow the DataNode to re-use existing
> threads, saving on the need to create and spin-up threads on demand.
> In this JIRA, I have added a couple of things:
> # Added a thread pool to {{DataXceiverServer}} class that, on demand, will
> create up to {{dfs.datanode.max.transfer.threads}}. A thread that has
> completed its prior duties will stay idle for up to 60 seconds
> (configurable), it will be retired if no new work has arrived.
> # Added new methods to the {{Peer}} Interface to allow for better logging and
> less code within each Thread ({{DataXceiver}}).
> # Updated the Thread code ({{DataXceiver}}) regarding its interactions with
> {{blockReceiver}} instance variable
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14292) Introduce Java ExecutorService to DataXceiverServer
[
https://issues.apache.org/jira/browse/HDFS-14292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-14292:
---
Attachment: HDFS-14292.7.patch
> Introduce Java ExecutorService to DataXceiverServer
> ---
>
> Key: HDFS-14292
> URL: https://issues.apache.org/jira/browse/HDFS-14292
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14292.1.patch, HDFS-14292.2.patch,
> HDFS-14292.3.patch, HDFS-14292.4.patch, HDFS-14292.5.patch,
> HDFS-14292.6.patch, HDFS-14292.6.patch, HDFS-14292.7.patch
>
>
> I wanted to investigate {{dfs.datanode.max.transfer.threads}} from
> {{hdfs-site.xml}}. It is described as "Specifies the maximum number of
> threads to use for transferring data in and out of the DN." The default
> value is 4096. I found it interesting because 4096 threads sounds like a lot
> to me. I'm not sure how a system with 8-16 cores would react to this large a
> thread count. Intuitively, I would say that the overhead of context
> switching would be immense.
> During mt investigation, I discovered the
> [following|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java#L203-L216]
> setup in the {{DataXceiverServer}} class:
> # A peer connects to a DataNode
> # A new thread is spun up to service this connection
> # The thread runs to completion
> # The tread dies
> It would perhaps be better if we used a thread pool to better manage the
> lifecycle of the service threads and to allow the DataNode to re-use existing
> threads, saving on the need to create and spin-up threads on demand.
> In this JIRA, I have added a couple of things:
> # Added a thread pool to {{DataXceiverServer}} class that, on demand, will
> create up to {{dfs.datanode.max.transfer.threads}}. A thread that has
> completed its prior duties will stay idle for up to 60 seconds
> (configurable), it will be retired if no new work has arrived.
> # Added new methods to the {{Peer}} Interface to allow for better logging and
> less code within each Thread ({{DataXceiver}}).
> # Updated the Thread code ({{DataXceiver}}) regarding its interactions with
> {{blockReceiver}} instance variable
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14292) Introduce Java ExecutorService to DataXceiverServer
[
https://issues.apache.org/jira/browse/HDFS-14292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-14292:
---
Status: Patch Available (was: Open)
> Introduce Java ExecutorService to DataXceiverServer
> ---
>
> Key: HDFS-14292
> URL: https://issues.apache.org/jira/browse/HDFS-14292
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14292.1.patch, HDFS-14292.2.patch,
> HDFS-14292.3.patch, HDFS-14292.4.patch, HDFS-14292.5.patch,
> HDFS-14292.6.patch, HDFS-14292.6.patch, HDFS-14292.7.patch
>
>
> I wanted to investigate {{dfs.datanode.max.transfer.threads}} from
> {{hdfs-site.xml}}. It is described as "Specifies the maximum number of
> threads to use for transferring data in and out of the DN." The default
> value is 4096. I found it interesting because 4096 threads sounds like a lot
> to me. I'm not sure how a system with 8-16 cores would react to this large a
> thread count. Intuitively, I would say that the overhead of context
> switching would be immense.
> During mt investigation, I discovered the
> [following|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java#L203-L216]
> setup in the {{DataXceiverServer}} class:
> # A peer connects to a DataNode
> # A new thread is spun up to service this connection
> # The thread runs to completion
> # The tread dies
> It would perhaps be better if we used a thread pool to better manage the
> lifecycle of the service threads and to allow the DataNode to re-use existing
> threads, saving on the need to create and spin-up threads on demand.
> In this JIRA, I have added a couple of things:
> # Added a thread pool to {{DataXceiverServer}} class that, on demand, will
> create up to {{dfs.datanode.max.transfer.threads}}. A thread that has
> completed its prior duties will stay idle for up to 60 seconds
> (configurable), it will be retired if no new work has arrived.
> # Added new methods to the {{Peer}} Interface to allow for better logging and
> less code within each Thread ({{DataXceiver}}).
> # Updated the Thread code ({{DataXceiver}}) regarding its interactions with
> {{blockReceiver}} instance variable
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14182) Datanode usage histogram is clicked to show ip list
[ https://issues.apache.org/jira/browse/HDFS-14182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16774676#comment-16774676 ] fengchuang commented on HDFS-14182: --- [~linyiqun] Hello,Can you help me to have a review again? thanks. > Datanode usage histogram is clicked to show ip list > --- > > Key: HDFS-14182 > URL: https://issues.apache.org/jira/browse/HDFS-14182 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: fengchuang >Assignee: fengchuang >Priority: Major > Attachments: HDFS-14182.001.patch, HDFS-14182.002.patch, > HDFS-14182.003.patch, showip.jpeg > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14307) RBF: Update tests to use internal Whitebox instead of Mockito
[ https://issues.apache.org/jira/browse/HDFS-14307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16774655#comment-16774655 ] CR Hota commented on HDFS-14307: [~elgoiri] [~ajisakaa] Thanks for the quick review. > RBF: Update tests to use internal Whitebox instead of Mockito > - > > Key: HDFS-14307 > URL: https://issues.apache.org/jira/browse/HDFS-14307 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: CR Hota >Assignee: CR Hota >Priority: Major > Fix For: HDFS-13891 > > Attachments: HDFS-14307-HDFS-13891.001.patch > > > Router tests should use internal whitebox instead of Mockito. Current router > feature development branch (HDFS-13891) fails without this. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1038) Support Service Level Authorization for OM, SCM and DN
[
https://issues.apache.org/jira/browse/HDDS-1038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16774667#comment-16774667
]
Xiaoyu Yao commented on HDDS-1038:
--
[~ajayydv], it is an scm security protocol but all messages are dealing with
scm CA certificates. Give the key has a prefix of security already, I think it
is better to avoid 2 security in the same config key here.
> Support Service Level Authorization for OM, SCM and DN
> --
>
> Key: HDDS-1038
> URL: https://issues.apache.org/jira/browse/HDDS-1038
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Ajay Kumar
>Assignee: Ajay Kumar
>Priority: Major
> Labels: Security
> Fix For: 0.4.0
>
> Attachments: HDDS-1038.00.patch, HDDS-1038.01.patch,
> HDDS-1038.02.patch, HDDS-1038.03.patch, HDDS-1038.04.patch,
> HDDS-1038.05.patch, HDDS-1038.06.patch, HDDS-1038.07.patch
>
>
> In a secure Ozone cluster. Datanodes fail to connect to SCM on
> {{StorageContainerDatanodeProtocol}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-14293) Increase Default Size of dfs.stream-buffer-size
[
https://issues.apache.org/jira/browse/HDFS-14293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16774669#comment-16774669
]
Shweta edited comment on HDFS-14293 at 2/22/19 1:44 AM:
[~belugabehr], since their occurrences/usages are less, I believe we can make
them deprecated and this way still can be used if need be.
was (Author: shwetayakkali):
Since their occurrences/usages are less, I believe we can make them deprecated
and this way still can be used if need be.
> Increase Default Size of dfs.stream-buffer-size
> ---
>
> Key: HDFS-14293
> URL: https://issues.apache.org/jira/browse/HDFS-14293
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Priority: Minor
>
> For many years (7+) now, the JDK has been using a default buffer size of
> [8192
> bytes|https://github.com/openjdk-mirror/jdk7u-jdk/blob/master/src/share/classes/java/io/BufferedInputStream.java#L53].
> Hadoop still defaults to a size half of that. The default is
> {{dfs.stream-buffer-size}} is 4096 bytes.
> https://hadoop.apache.org/docs/r2.8.0/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml
> Please increase default size to 8192.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14305) Serial number in BlockTokenSecretManager could overlap between different namenodes
[
https://issues.apache.org/jira/browse/HDFS-14305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16774662#comment-16774662
]
Wei-Chiu Chuang commented on HDFS-14305:
Is this a problem when you have observer nodes? I.e. standby NameNode doesn't
issue block tokens so we didn't see it before?
> Serial number in BlockTokenSecretManager could overlap between different
> namenodes
> --
>
> Key: HDFS-14305
> URL: https://issues.apache.org/jira/browse/HDFS-14305
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: security
>Reporter: Chao Sun
>Assignee: Chao Sun
>Priority: Major
>
> Currently, a {{BlockTokenSecretManager}} starts with a random integer as the
> initial serial number, and then use this formula to rotate it:
> {code:java}
> this.intRange = Integer.MAX_VALUE / numNNs;
> this.nnRangeStart = intRange * nnIndex;
> this.serialNo = (this.serialNo % intRange) + (nnRangeStart);
> {code}
> while {{numNNs}} is the total number of NameNodes in the cluster, and
> {{nnIndex}} is the index of the current NameNode specified in the
> configuration {{dfs.ha.namenodes.}}.
> However, with this approach, different NameNode could have overlapping ranges
> for serial number. For simplicity, let's assume {{Integer.MAX_VALUE}} is 100,
> and we have 2 NameNodes {{nn1}} and {{nn2}} in configuration. Then the ranges
> for these two are:
> {code}
> nn1 -> [-49, 49]
> nn2 -> [1, 99]
> {code}
> This is because the initial serial number could be any negative integer.
> Moreover, when the keys are updated, the serial number will again be updated
> with the formula:
> {code}
> this.serialNo = (this.serialNo % intRange) + (nnRangeStart);
> {code}
> which means the new serial number could be updated to a range that belongs to
> a different NameNode, thus increasing the chance of collision again.
> When the collision happens, DataNodes could overwrite an existing key which
> will cause clients to fail because of {{InvalidToken}} error.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14293) Increase Default Size of dfs.stream-buffer-size
[
https://issues.apache.org/jira/browse/HDFS-14293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16774669#comment-16774669
]
Shweta commented on HDFS-14293:
---
Since their occurrences/usages are less, I believe we can make them deprecated
and this way still can be used if need be.
> Increase Default Size of dfs.stream-buffer-size
> ---
>
> Key: HDFS-14293
> URL: https://issues.apache.org/jira/browse/HDFS-14293
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Priority: Minor
>
> For many years (7+) now, the JDK has been using a default buffer size of
> [8192
> bytes|https://github.com/openjdk-mirror/jdk7u-jdk/blob/master/src/share/classes/java/io/BufferedInputStream.java#L53].
> Hadoop still defaults to a size half of that. The default is
> {{dfs.stream-buffer-size}} is 4096 bytes.
> https://hadoop.apache.org/docs/r2.8.0/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml
> Please increase default size to 8192.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1061) DelegationToken: Add certificate serial id to Ozone Delegation Token Identifier
[ https://issues.apache.org/jira/browse/HDDS-1061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16774663#comment-16774663 ] Xiaoyu Yao commented on HDDS-1061: -- +1 for v3 patch pending Jenkins. > DelegationToken: Add certificate serial id to Ozone Delegation Token > Identifier > > > Key: HDDS-1061 > URL: https://issues.apache.org/jira/browse/HDDS-1061 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Attachments: HDDS-1061.00.patch, HDDS-1061.01.patch, > HDDS-1061.02.patch, HDDS-1061.03.patch > > > 1. Add certificate serial id to Ozone Delegation Token Identifier. Required > for OM HA support. > 2. Validate Ozone token based on public key from OM certificate -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14295) Add Threadpool for DataTransfers
[
https://issues.apache.org/jira/browse/HDFS-14295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16774659#comment-16774659
]
Íñigo Goiri commented on HDFS-14295:
OK let's go with the cast and let's add a supresswarning and so on.
What about unit tests for this?
> Add Threadpool for DataTransfers
>
>
> Key: HDFS-14295
> URL: https://issues.apache.org/jira/browse/HDFS-14295
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14295.1.patch, HDFS-14295.2.patch,
> HDFS-14295.3.patch, HDFS-14295.4.patch, HDFS-14295.5.patch
>
>
> When a DataNode data transfers a block, is spins up a new thread for each
> transfer.
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339]
> and
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022].
> Instead, add the threads to a {{CachedThreadPool}} so that when their
> threads complete the transfer, they can be re-used for another transfer. This
> should save resources spent on creating and spinning up transfer threads.
> One thing I'll point out that's a bit off, which I address in this patch, ...
> There are two places in the code where a {{DataTransfer}} thread is started.
> In [one
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339-L2341],
> it's started in a default thread group. In [another
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022],
> it's started in the
> [dataXceiverServer|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L1164]
> thread group.
> I do not think it's correct to include any of these threads in the
> {{dataXceiverServer}} thread group. Anything submitted to the
> {{dataXceiverServer}} should probably be tied to the
> {{dfs.datanode.max.transfer.threads}} configurations, and neither of these
> methods are. Instead, they should be submitted into the same thread pool with
> its own thread group (probably the default thread group, unless someone
> suggests otherwise) and is what I have included in this patch.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14118) Use DNS to resolve Namenodes and Routers
[
https://issues.apache.org/jira/browse/HDFS-14118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16774653#comment-16774653
]
Hadoop QA commented on HDFS-14118:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
3s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 11s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
41s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 43s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
8s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
55s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 79m 10s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
46s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}196m 57s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes |
| | hadoop.hdfs.qjournal.server.TestJournalNodeSync |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-14118 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12959679/HDFS-14118.020.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite
[jira] [Commented] (HDFS-14052) RBF: Use Router keytab for WebHDFS
[
https://issues.apache.org/jira/browse/HDFS-14052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16774652#comment-16774652
]
Íñigo Goiri commented on HDFS-14052:
Thanks [~crh] for opening the patch.
HDFS-14307 it is already committed to the branch so the run should be clean now.
> RBF: Use Router keytab for WebHDFS
> --
>
> Key: HDFS-14052
> URL: https://issues.apache.org/jira/browse/HDFS-14052
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: CR Hota
>Priority: Major
> Attachments: HDFS-14052-HDFS-13891.0.patch,
> HDFS-14052-HDFS-13891.1.patch
>
>
> When the RouterHttpServer starts it does:
> {code}
> NameNodeHttpServer.initWebHdfs(conf, httpAddress.getHostName(),
> httpServer,
> RouterWebHdfsMethods.class.getPackage().getName());
> {code}
> This function is in the NN and is pretty generic.
> However, it then calls to NameNodeHttpServer#getAuthFilterParams, which does:
> {code}
> String httpKeytab = conf.get(DFSUtil.getSpnegoKeytabKey(conf,
> DFSConfigKeys.DFS_NAMENODE_KEYTAB_FILE_KEY));
> {code}
> In most cases, the regular web keytab will kick in, but we should make this a
> parameter and load the Router one just in case.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14307) RBF: Update tests to use internal Whitebox instead of Mockito
[ https://issues.apache.org/jira/browse/HDFS-14307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-14307: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: HDFS-13891 Status: Resolved (was: Patch Available) > RBF: Update tests to use internal Whitebox instead of Mockito > - > > Key: HDFS-14307 > URL: https://issues.apache.org/jira/browse/HDFS-14307 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: CR Hota >Assignee: CR Hota >Priority: Major > Fix For: HDFS-13891 > > Attachments: HDFS-14307-HDFS-13891.001.patch > > > Router tests should use internal whitebox instead of Mockito. Current router > feature development branch (HDFS-13891) fails without this. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14307) RBF: Update tests to use internal Whitebox instead of Mockito
[ https://issues.apache.org/jira/browse/HDFS-14307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16774651#comment-16774651 ] Íñigo Goiri commented on HDFS-14307: Thanks [~crh] for the patch and [~ajisakaa] for the review. Committed to HDFS-13891; it should compile now. > RBF: Update tests to use internal Whitebox instead of Mockito > - > > Key: HDFS-14307 > URL: https://issues.apache.org/jira/browse/HDFS-14307 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: CR Hota >Assignee: CR Hota >Priority: Major > Fix For: HDFS-13891 > > Attachments: HDFS-14307-HDFS-13891.001.patch > > > Router tests should use internal whitebox instead of Mockito. Current router > feature development branch (HDFS-13891) fails without this. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14307) RBF: Update tests to use internal Whitebox instead of Mockito
[ https://issues.apache.org/jira/browse/HDFS-14307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16774648#comment-16774648 ] Íñigo Goiri commented on HDFS-14307: The javac warning says that Whitebox has been deprecated. We can try to get out of it in a followup JIRA. For now I'm committing this. +1 on [^HDFS-14307-HDFS-13891.001.patch]. > RBF: Update tests to use internal Whitebox instead of Mockito > - > > Key: HDFS-14307 > URL: https://issues.apache.org/jira/browse/HDFS-14307 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: CR Hota >Assignee: CR Hota >Priority: Major > Attachments: HDFS-14307-HDFS-13891.001.patch > > > Router tests should use internal whitebox instead of Mockito. Current router > feature development branch (HDFS-13891) fails without this. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-14285) libhdfs hdfsRead copies entire array even if its only partially filled
[
https://issues.apache.org/jira/browse/HDFS-14285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16774642#comment-16774642
]
Wei-Chiu Chuang edited comment on HDFS-14285 at 2/22/19 1:09 AM:
-
Patch makes sense to me. This is exactly what hdfsPread() does.
was (Author: jojochuang):
Patch makes sense to me.
> libhdfs hdfsRead copies entire array even if its only partially filled
> --
>
> Key: HDFS-14285
> URL: https://issues.apache.org/jira/browse/HDFS-14285
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client, libhdfs, native
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HDFS-14285.001.patch, HDFS-14285.002.patch
>
>
> There is a bug in libhdfs {{hdfsRead}}
> {code:java}
> jthr = invokeMethod(env, , INSTANCE, jInputStream, HADOOP_ISTRM,
>"read", "([B)I", jbRarray);
> if (jthr) {
> destroyLocalReference(env, jbRarray);
> errno = printExceptionAndFree(env, jthr, PRINT_EXC_ALL,
> "hdfsRead: FSDataInputStream#read");
> return -1;
> }
> if (jVal.i < 0) {
> // EOF
> destroyLocalReference(env, jbRarray);
> return 0;
> } else if (jVal.i == 0) {
> destroyLocalReference(env, jbRarray);
> errno = EINTR;
> return -1;
> }
> (*env)->GetByteArrayRegion(env, jbRarray, 0, noReadBytes, buffer);
> {code}
> The method makes a call to {{FSInputStream#read(byte[])}} to fill in the Java
> byte array, however, {{#read(byte[])}} is not guaranteed to fill up the
> entire array, instead it returns the number of bytes written to the array
> (which could be less than the size of the array). Yet `{{GetByteArrayRegion}}
> decides to copy the entire contents of the {{jbArray}} into the buffer
> ({{noReadBytes}} is initialized to the length of the buffer and is never
> updated). So if {{FSInputStream#read(byte[])}} decides to read less data than
> the size of the byte array, the call to {{GetByteArrayRegion}} will
> essentially copy more bytes than necessary.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14285) libhdfs hdfsRead copies entire array even if its only partially filled
[
https://issues.apache.org/jira/browse/HDFS-14285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16774642#comment-16774642
]
Wei-Chiu Chuang commented on HDFS-14285:
Patch makes sense to me.
> libhdfs hdfsRead copies entire array even if its only partially filled
> --
>
> Key: HDFS-14285
> URL: https://issues.apache.org/jira/browse/HDFS-14285
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client, libhdfs, native
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HDFS-14285.001.patch, HDFS-14285.002.patch
>
>
> There is a bug in libhdfs {{hdfsRead}}
> {code:java}
> jthr = invokeMethod(env, , INSTANCE, jInputStream, HADOOP_ISTRM,
>"read", "([B)I", jbRarray);
> if (jthr) {
> destroyLocalReference(env, jbRarray);
> errno = printExceptionAndFree(env, jthr, PRINT_EXC_ALL,
> "hdfsRead: FSDataInputStream#read");
> return -1;
> }
> if (jVal.i < 0) {
> // EOF
> destroyLocalReference(env, jbRarray);
> return 0;
> } else if (jVal.i == 0) {
> destroyLocalReference(env, jbRarray);
> errno = EINTR;
> return -1;
> }
> (*env)->GetByteArrayRegion(env, jbRarray, 0, noReadBytes, buffer);
> {code}
> The method makes a call to {{FSInputStream#read(byte[])}} to fill in the Java
> byte array, however, {{#read(byte[])}} is not guaranteed to fill up the
> entire array, instead it returns the number of bytes written to the array
> (which could be less than the size of the array). Yet `{{GetByteArrayRegion}}
> decides to copy the entire contents of the {{jbArray}} into the buffer
> ({{noReadBytes}} is initialized to the length of the buffer and is never
> updated). So if {{FSInputStream#read(byte[])}} decides to read less data than
> the size of the byte array, the call to {{GetByteArrayRegion}} will
> essentially copy more bytes than necessary.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14307) RBF: Update tests to use internal Whitebox instead of Mockito
[ https://issues.apache.org/jira/browse/HDFS-14307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16774639#comment-16774639 ] Akira Ajisaka commented on HDFS-14307: -- +1, thanks [~crh]. > RBF: Update tests to use internal Whitebox instead of Mockito > - > > Key: HDFS-14307 > URL: https://issues.apache.org/jira/browse/HDFS-14307 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: CR Hota >Assignee: CR Hota >Priority: Major > Attachments: HDFS-14307-HDFS-13891.001.patch > > > Router tests should use internal whitebox instead of Mockito. Current router > feature development branch (HDFS-13891) fails without this. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14309) name node fail over failed with ssh fence failed because of jsch login failed with key check
[
https://issues.apache.org/jira/browse/HDFS-14309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16774634#comment-16774634
]
Hadoop QA commented on HDFS-14309:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 2m
40s{color} | {color:red} Docker failed to build yetus/hadoop:c420dfe. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-14309 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12959692/HDFS-14309-branch-2.7.3.001.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/26298/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> name node fail over failed with ssh fence failed because of jsch login failed
> with key check
>
>
> Key: HDFS-14309
> URL: https://issues.apache.org/jira/browse/HDFS-14309
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: auto-failover
>Affects Versions: 2.7.3
> Environment: linux CentOS release 6.8 (Final) kernel
> 2.6.32-642.6.1.el6.x86_64
>
>Reporter: qiang Liu
>Priority: Major
> Attachments: HDFS-14309-branch-2.7.3.001.patch
>
> Original Estimate: 1m
> Remaining Estimate: 1m
>
> name node fail over failed with ssh fence failed because of jsch login failed
> with key check.
> the loged error is "Algorithm negotiation fail"
> update jsch to 0.1.54 works ok
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14307) RBF: Update tests to use internal Whitebox instead of Mockito
[
https://issues.apache.org/jira/browse/HDFS-14307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16774631#comment-16774631
]
Hadoop QA commented on HDFS-14307:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-13891 Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 16m
5s{color} | {color:red} root in HDFS-13891 failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
24s{color} | {color:red} hadoop-hdfs-rbf in HDFS-13891 failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} HDFS-13891 passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
29s{color} | {color:red} hadoop-hdfs-rbf in HDFS-13891 failed. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 42s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
27s{color} | {color:red} hadoop-hdfs-rbf in HDFS-13891 failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
32s{color} | {color:green} HDFS-13891 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 25s{color}
| {color:red} hadoop-hdfs-project_hadoop-hdfs-rbf generated 3 new + 9 unchanged
- 12 fixed = 12 total (was 21) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 32s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 22m
32s{color} | {color:green} hadoop-hdfs-rbf in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 67m 16s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-14307 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12959684/HDFS-14307-HDFS-13891.001.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux dff5e5c0b185 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | HDFS-13891 / f476bb1 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| mvninstall |
https://builds.apache.org/job/PreCommit-HDFS-Build/26294/artifact/out/branch-mvninstall-root.txt
|
| compile |
https://builds.apache.org/job/PreCommit-HDFS-Build/26294/artifact/out/branch-compile-hadoop-hdfs-project_hadoop-hdfs-rbf.txt
|
| mvnsite |
https://builds.apache.org/job/PreCommit-HDFS-Build/26294/artifact/out/branch-mvnsite-hadoop-hdfs-project_hadoop-hdfs-rbf.txt
|
| findbugs |
[jira] [Work logged] (HDDS-919) Enable prometheus endpoints for Ozone datanodes
[ https://issues.apache.org/jira/browse/HDDS-919?focusedWorklogId=202308=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-202308 ] ASF GitHub Bot logged work on HDDS-919: --- Author: ASF GitHub Bot Created on: 22/Feb/19 00:46 Start Date: 22/Feb/19 00:46 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on issue #502: HDDS-919. Enable prometheus endpoints for Ozone datanodes URL: https://github.com/apache/hadoop/pull/502#issuecomment-466228031 @elek Can you rebase this PR? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 202308) Time Spent: 0.5h (was: 20m) > Enable prometheus endpoints for Ozone datanodes > --- > > Key: HDDS-919 > URL: https://issues.apache.org/jira/browse/HDDS-919 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Labels: newbie, pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > HDDS-846 provides a new metric endpoint which publishes the available Hadoop > metrics in prometheus friendly format with a new servlet. > Unfortunately it's enabled only on the scm/om side. It would be great to > enable it in the Ozone/HDDS datanodes on the web server of the HDDS Rest > endpoint. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-919) Enable prometheus endpoints for Ozone datanodes
[ https://issues.apache.org/jira/browse/HDDS-919?focusedWorklogId=202307=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-202307 ] ASF GitHub Bot logged work on HDDS-919: --- Author: ASF GitHub Bot Created on: 22/Feb/19 00:45 Start Date: 22/Feb/19 00:45 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on issue #502: HDDS-919. Enable prometheus endpoints for Ozone datanodes URL: https://github.com/apache/hadoop/pull/502#issuecomment-466228031 @elek Overall patch LGTM. This needs to be rebased to merge. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 202307) Time Spent: 20m (was: 10m) > Enable prometheus endpoints for Ozone datanodes > --- > > Key: HDDS-919 > URL: https://issues.apache.org/jira/browse/HDDS-919 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Labels: newbie, pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > HDDS-846 provides a new metric endpoint which publishes the available Hadoop > metrics in prometheus friendly format with a new servlet. > Unfortunately it's enabled only on the scm/om side. It would be great to > enable it in the Ozone/HDDS datanodes on the web server of the HDDS Rest > endpoint. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14303) chek block directory logic not correct when there is only meta file, print no meaning warn log
[
https://issues.apache.org/jira/browse/HDFS-14303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16774628#comment-16774628
]
Hadoop QA commented on HDFS-14303:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 3m
0s{color} | {color:red} Docker failed to build yetus/hadoop:c420dfe. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-14303 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12959690/HDFS-14303-branch-2.7.3.002.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/26296/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> chek block directory logic not correct when there is only meta file, print no
> meaning warn log
> --
>
> Key: HDFS-14303
> URL: https://issues.apache.org/jira/browse/HDFS-14303
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, hdfs
>Affects Versions: 2.7.3
> Environment: env free
>Reporter: qiang Liu
>Priority: Minor
> Labels: easy-fix
> Attachments: HDFS-14303-branch-2.7.001.patch,
> HDFS-14303-branch-2.7.001.patch, HDFS-14303-branch-2.7.3.002.patch
>
> Original Estimate: 1m
> Remaining Estimate: 1m
>
> chek block directory logic not correct when there is only meta file,print no
> meaning warn log, eg:
> WARN DirectoryScanner:? - Block: 1101939874 has to be upgraded to block
> ID-based layout. Actual block file path:
> /data14/hadoop/data/current/BP-1461038173-10.8.48.152-1481686842620/current/finalized/subdir174/subdir68,
> expected block file path:
> /data14/hadoop/data/current/BP-1461038173-10.8.48.152-1481686842620/current/finalized/subdir174/subdir68/subdir68
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1113) Remove default dependencies from hadoop-ozone project
[ https://issues.apache.org/jira/browse/HDDS-1113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16774627#comment-16774627 ] Bharat Viswanadham commented on HDDS-1113: -- Hi [~elek] Thanks for the contribution. Build compilation is failing with this patch. > Remove default dependencies from hadoop-ozone project > - > > Key: HDDS-1113 > URL: https://issues.apache.org/jira/browse/HDDS-1113 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: build >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > There are two ways to define common dependencies with maven: > 1.) put all the dependencies to the parent project and inherit them > 2.) get all the dependencies via transitive dependencies > TLDR; I would like to switch from 1 to 2 in hadoop-ozone > My main problem with the first approach that all the child project get a lot > of dependencies independent if they need them or not. Let's imagine that I > would like to create a new project (for example a java csi implementation) It > doesn't need ozone-client, ozone-common etc, in fact it conflicts with > ozone-client. But these jars are always added as of now. > Using transitive dependencies is more safe: we can add the dependencies where > we need them and all of the other dependent projects will use them. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14295) Add Threadpool for DataTransfers
[
https://issues.apache.org/jira/browse/HDFS-14295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-14295:
---
Status: Patch Available (was: Open)
Ya, [~elgoiri]. I think we're better off with the cast into a Future and then
someday, when the API is fixed, the cast can simply be removed.
> Add Threadpool for DataTransfers
>
>
> Key: HDFS-14295
> URL: https://issues.apache.org/jira/browse/HDFS-14295
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14295.1.patch, HDFS-14295.2.patch,
> HDFS-14295.3.patch, HDFS-14295.4.patch, HDFS-14295.5.patch
>
>
> When a DataNode data transfers a block, is spins up a new thread for each
> transfer.
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339]
> and
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022].
> Instead, add the threads to a {{CachedThreadPool}} so that when their
> threads complete the transfer, they can be re-used for another transfer. This
> should save resources spent on creating and spinning up transfer threads.
> One thing I'll point out that's a bit off, which I address in this patch, ...
> There are two places in the code where a {{DataTransfer}} thread is started.
> In [one
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339-L2341],
> it's started in a default thread group. In [another
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022],
> it's started in the
> [dataXceiverServer|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L1164]
> thread group.
> I do not think it's correct to include any of these threads in the
> {{dataXceiverServer}} thread group. Anything submitted to the
> {{dataXceiverServer}} should probably be tied to the
> {{dfs.datanode.max.transfer.threads}} configurations, and neither of these
> methods are. Instead, they should be submitted into the same thread pool with
> its own thread group (probably the default thread group, unless someone
> suggests otherwise) and is what I have included in this patch.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14309) name node fail over failed with ssh fence failed because of jsch login failed with key check
[ https://issues.apache.org/jira/browse/HDFS-14309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qiang Liu updated HDFS-14309: - Attachment: HDFS-14309-branch-2.7.3.001.patch Status: Patch Available (was: Open) > name node fail over failed with ssh fence failed because of jsch login failed > with key check > > > Key: HDFS-14309 > URL: https://issues.apache.org/jira/browse/HDFS-14309 > Project: Hadoop HDFS > Issue Type: Bug > Components: auto-failover >Affects Versions: 2.7.3 > Environment: linux CentOS release 6.8 (Final) kernel > 2.6.32-642.6.1.el6.x86_64 > >Reporter: qiang Liu >Priority: Major > Attachments: HDFS-14309-branch-2.7.3.001.patch > > Original Estimate: 1m > Remaining Estimate: 1m > > name node fail over failed with ssh fence failed because of jsch login failed > with key check. > the loged error is "Algorithm negotiation fail" > update jsch to 0.1.54 works ok -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-14309) name node fail over failed with ssh fence failed because of jsch login failed with key check
qiang Liu created HDFS-14309: Summary: name node fail over failed with ssh fence failed because of jsch login failed with key check Key: HDFS-14309 URL: https://issues.apache.org/jira/browse/HDFS-14309 Project: Hadoop HDFS Issue Type: Bug Components: auto-failover Affects Versions: 2.7.3 Environment: linux CentOS release 6.8 (Final) kernel 2.6.32-642.6.1.el6.x86_64 Reporter: qiang Liu name node fail over failed with ssh fence failed because of jsch login failed with key check. the loged error is "Algorithm negotiation fail" update jsch to 0.1.54 works ok -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14303) chek block directory logic not correct when there is only meta file, print no meaning warn log
[
https://issues.apache.org/jira/browse/HDFS-14303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16774615#comment-16774615
]
Hadoop QA commented on HDFS-14303:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 12m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} branch-2.7 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
9s{color} | {color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
16s{color} | {color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s{color} | {color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
4s{color} | {color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
53s{color} | {color:green} branch-2.7 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 60 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
4s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}706m 49s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 2m
12s{color} | {color:red} The patch generated 14 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}743m 50s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Unreaped Processes | hadoop-hdfs:44 |
| Failed junit tests | hadoop.hdfs.TestEncryptionZonesWithHA |
| | hadoop.hdfs.qjournal.TestMiniJournalCluster |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.TestFileCorruption |
| Timed out junit tests | org.apache.hadoop.hdfs.TestEncryptionZones |
| | org.apache.hadoop.hdfs.tools.TestDFSAdminWithHA |
| | org.apache.hadoop.hdfs.qjournal.client.TestQuorumJournalManager |
| | org.apache.hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer |
| | org.apache.hadoop.hdfs.TestDatanodeRegistration |
| | org.apache.hadoop.hdfs.TestSetrepDecreasing |
| | org.apache.hadoop.hdfs.TestReplaceDatanodeOnFailure |
| | org.apache.hadoop.hdfs.qjournal.server.TestJournalNode |
| | org.apache.hadoop.hdfs.TestQuota |
| | org.apache.hadoop.hdfs.TestDataTransferKeepalive |
| | org.apache.hadoop.hdfs.TestDatanodeDeath |
| | org.apache.hadoop.hdfs.TestFileLengthOnClusterRestart |
| | org.apache.hadoop.hdfs.TestFileAppend |
| | org.apache.hadoop.hdfs.TestPread |
| | org.apache.hadoop.hdfs.TestFileAppend4 |
| | org.apache.hadoop.hdfs.TestDFSFinalize |
| | org.apache.hadoop.hdfs.qjournal.client.TestQJMWithFaults |
| | org.apache.hadoop.hdfs.TestDFSUpgradeFromImage |
| | org.apache.hadoop.hdfs.web.TestWebHdfsTokens |
| | org.apache.hadoop.hdfs.TestDatanodeStartupFixesLegacyStorageIDs |
| | org.apache.hadoop.hdfs.TestEncryptionZonesWithKMS |
| | org.apache.hadoop.hdfs.protocol.datatransfer.sasl.TestSaslDataTransfer |
| | org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM |
| |
[jira] [Commented] (HDFS-14303) chek block directory logic not correct when there is only meta file, print no meaning warn log
[ https://issues.apache.org/jira/browse/HDFS-14303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16774621#comment-16774621 ] qiang Liu commented on HDFS-14303: -- patch for branch 2.7.3 > chek block directory logic not correct when there is only meta file, print no > meaning warn log > -- > > Key: HDFS-14303 > URL: https://issues.apache.org/jira/browse/HDFS-14303 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, hdfs >Affects Versions: 2.7.3 > Environment: env free >Reporter: qiang Liu >Priority: Minor > Labels: easy-fix > Attachments: HDFS-14303-branch-2.7.001.patch, > HDFS-14303-branch-2.7.001.patch, HDFS-14303-branch-2.7.3.002.patch > > Original Estimate: 1m > Remaining Estimate: 1m > > chek block directory logic not correct when there is only meta file,print no > meaning warn log, eg: > WARN DirectoryScanner:? - Block: 1101939874 has to be upgraded to block > ID-based layout. Actual block file path: > /data14/hadoop/data/current/BP-1461038173-10.8.48.152-1481686842620/current/finalized/subdir174/subdir68, > expected block file path: > /data14/hadoop/data/current/BP-1461038173-10.8.48.152-1481686842620/current/finalized/subdir174/subdir68/subdir68 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14295) Add Threadpool for DataTransfers
[
https://issues.apache.org/jira/browse/HDFS-14295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-14295:
---
Attachment: HDFS-14295.5.patch
> Add Threadpool for DataTransfers
>
>
> Key: HDFS-14295
> URL: https://issues.apache.org/jira/browse/HDFS-14295
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14295.1.patch, HDFS-14295.2.patch,
> HDFS-14295.3.patch, HDFS-14295.4.patch, HDFS-14295.5.patch
>
>
> When a DataNode data transfers a block, is spins up a new thread for each
> transfer.
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339]
> and
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022].
> Instead, add the threads to a {{CachedThreadPool}} so that when their
> threads complete the transfer, they can be re-used for another transfer. This
> should save resources spent on creating and spinning up transfer threads.
> One thing I'll point out that's a bit off, which I address in this patch, ...
> There are two places in the code where a {{DataTransfer}} thread is started.
> In [one
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339-L2341],
> it's started in a default thread group. In [another
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022],
> it's started in the
> [dataXceiverServer|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L1164]
> thread group.
> I do not think it's correct to include any of these threads in the
> {{dataXceiverServer}} thread group. Anything submitted to the
> {{dataXceiverServer}} should probably be tied to the
> {{dfs.datanode.max.transfer.threads}} configurations, and neither of these
> methods are. Instead, they should be submitted into the same thread pool with
> its own thread group (probably the default thread group, unless someone
> suggests otherwise) and is what I have included in this patch.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14295) Add Threadpool for DataTransfers
[
https://issues.apache.org/jira/browse/HDFS-14295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-14295:
---
Status: Open (was: Patch Available)
> Add Threadpool for DataTransfers
>
>
> Key: HDFS-14295
> URL: https://issues.apache.org/jira/browse/HDFS-14295
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14295.1.patch, HDFS-14295.2.patch,
> HDFS-14295.3.patch, HDFS-14295.4.patch, HDFS-14295.5.patch
>
>
> When a DataNode data transfers a block, is spins up a new thread for each
> transfer.
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339]
> and
> [Here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022].
> Instead, add the threads to a {{CachedThreadPool}} so that when their
> threads complete the transfer, they can be re-used for another transfer. This
> should save resources spent on creating and spinning up transfer threads.
> One thing I'll point out that's a bit off, which I address in this patch, ...
> There are two places in the code where a {{DataTransfer}} thread is started.
> In [one
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L2339-L2341],
> it's started in a default thread group. In [another
> place|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3019-L3022],
> it's started in the
> [dataXceiverServer|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L1164]
> thread group.
> I do not think it's correct to include any of these threads in the
> {{dataXceiverServer}} thread group. Anything submitted to the
> {{dataXceiverServer}} should probably be tied to the
> {{dfs.datanode.max.transfer.threads}} configurations, and neither of these
> methods are. Instead, they should be submitted into the same thread pool with
> its own thread group (probably the default thread group, unless someone
> suggests otherwise) and is what I have included in this patch.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-14303) chek block directory logic not correct when there is only meta file, print no meaning warn log
[ https://issues.apache.org/jira/browse/HDFS-14303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16774621#comment-16774621 ] qiang Liu edited comment on HDFS-14303 at 2/22/19 12:33 AM: patch for branch 2.7.3 [^HDFS-14303-branch-2.7.3.002.patch] was (Author: iamgd67): patch for branch 2.7.3 > chek block directory logic not correct when there is only meta file, print no > meaning warn log > -- > > Key: HDFS-14303 > URL: https://issues.apache.org/jira/browse/HDFS-14303 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, hdfs >Affects Versions: 2.7.3 > Environment: env free >Reporter: qiang Liu >Priority: Minor > Labels: easy-fix > Attachments: HDFS-14303-branch-2.7.001.patch, > HDFS-14303-branch-2.7.001.patch, HDFS-14303-branch-2.7.3.002.patch > > Original Estimate: 1m > Remaining Estimate: 1m > > chek block directory logic not correct when there is only meta file,print no > meaning warn log, eg: > WARN DirectoryScanner:? - Block: 1101939874 has to be upgraded to block > ID-based layout. Actual block file path: > /data14/hadoop/data/current/BP-1461038173-10.8.48.152-1481686842620/current/finalized/subdir174/subdir68, > expected block file path: > /data14/hadoop/data/current/BP-1461038173-10.8.48.152-1481686842620/current/finalized/subdir174/subdir68/subdir68 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14303) chek block directory logic not correct when there is only meta file, print no meaning warn log
[ https://issues.apache.org/jira/browse/HDFS-14303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qiang Liu updated HDFS-14303: - Attachment: HDFS-14303-branch-2.7.3.002.patch > chek block directory logic not correct when there is only meta file, print no > meaning warn log > -- > > Key: HDFS-14303 > URL: https://issues.apache.org/jira/browse/HDFS-14303 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, hdfs >Affects Versions: 2.7.3 > Environment: env free >Reporter: qiang Liu >Priority: Minor > Labels: easy-fix > Attachments: HDFS-14303-branch-2.7.001.patch, > HDFS-14303-branch-2.7.001.patch, HDFS-14303-branch-2.7.3.002.patch > > Original Estimate: 1m > Remaining Estimate: 1m > > chek block directory logic not correct when there is only meta file,print no > meaning warn log, eg: > WARN DirectoryScanner:? - Block: 1101939874 has to be upgraded to block > ID-based layout. Actual block file path: > /data14/hadoop/data/current/BP-1461038173-10.8.48.152-1481686842620/current/finalized/subdir174/subdir68, > expected block file path: > /data14/hadoop/data/current/BP-1461038173-10.8.48.152-1481686842620/current/finalized/subdir174/subdir68/subdir68 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14308) DFSStripedInputStream should implement unbuffer()
[
https://issues.apache.org/jira/browse/HDFS-14308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16774616#comment-16774616
]
Wei-Chiu Chuang commented on HDFS-14308:
[~knanasi] could you take a look?
> DFSStripedInputStream should implement unbuffer()
> -
>
> Key: HDFS-14308
> URL: https://issues.apache.org/jira/browse/HDFS-14308
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Joe McDonnell
>Priority: Major
> Attachments: ec_heap_dump.png
>
>
> Some users of HDFS cache opened HDFS file handles to avoid repeated
> roundtrips to the NameNode. For example, Impala caches up to 20,000 HDFS file
> handles by default. Recent tests on erasure coded files show that the open
> file handles can consume a large amount of memory when not in use.
> For example, here is output from Impala's JMX endpoint when 608 file handles
> are cached
> {noformat}
> {
> "name": "java.nio:type=BufferPool,name=direct",
> "modelerType": "sun.management.ManagementFactoryHelper$1",
> "Name": "direct",
> "TotalCapacity": 1921048960,
> "MemoryUsed": 1921048961,
> "Count": 633,
> "ObjectName": "java.nio:type=BufferPool,name=direct"
> },{noformat}
> This shows direct buffer memory usage of 3MB per DFSStripedInputStream.
> Attached is output from Eclipse MAT showing that the direct buffers come from
> DFSStripedInputStream objects.
> To support caching file handles on erasure coded files, DFSStripedInputStream
> should implement the unbuffer() call. See HDFS-7694. "unbuffer()" is intended
> to move an input stream to a lower memory state to support these caching use
> cases. Both Impala and HBase call unbuffer() when a file handle is being
> cached and potentially unused for significant chunks of time.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14130) Make ZKFC ObserverNode aware
[ https://issues.apache.org/jira/browse/HDFS-14130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated HDFS-14130: --- Attachment: HDFS-14130.010.patch > Make ZKFC ObserverNode aware > > > Key: HDFS-14130 > URL: https://issues.apache.org/jira/browse/HDFS-14130 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha >Affects Versions: HDFS-12943 >Reporter: Konstantin Shvachko >Assignee: xiangheng >Priority: Major > Attachments: HDFS-14130-HDFS-12943.001.patch, > HDFS-14130-HDFS-12943.003.patch, HDFS-14130-HDFS-12943.004.patch, > HDFS-14130-HDFS-12943.005.patch, HDFS-14130-HDFS-12943.006.patch, > HDFS-14130-HDFS-12943.007.patch, HDFS-14130.008.patch, HDFS-14130.009.patch, > HDFS-14130.010.patch > > > Need to fix automatic failover with ZKFC. Currently it does not know about > ObserverNodes trying to convert them to SBNs. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-3246) pRead equivalent for direct read path
[
https://issues.apache.org/jira/browse/HDFS-3246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16774611#comment-16774611
]
Hadoop QA commented on HDFS-3246:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
12s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
18m 25s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-hdfs-project/hadoop-hdfs-native-client {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
14s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
2s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 14m 2s{color} |
{color:red} root generated 2 new + 9 unchanged - 0 fixed = 11 total (was 9)
{color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
58s{color} | {color:green} root: The patch generated 0 new + 48 unchanged - 1
fixed = 48 total (was 49) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 44s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-hdfs-project/hadoop-hdfs-native-client {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
39s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
1s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 75m 52s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 5m
47s{color} | {color:green} hadoop-hdfs-native-client in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
54s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}200m
[jira] [Comment Edited] (HDDS-1043) Enable token based authentication for S3 api
[ https://issues.apache.org/jira/browse/HDDS-1043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16774609#comment-16774609 ] Bharat Viswanadham edited comment on HDDS-1043 at 2/22/19 12:02 AM: Hi [~ajayydv] Few comments I have: # Can we use already existing Header parsers AuthorizationHeaderV4 and AuthorizationHeaderV2.java instead of parsing it again in new class AWSV4AuthParser. Same comment for V2 parser. And also can we added reference links, so that it will be easy to refer aws header documentation. # And also we have AuthenticationHeaderParser which checks type V2 and V4. And then do required. I think we should do similar checks in OzoneClientProducer.java and then create token? # In OzoneDelegationTokenSecretManager.java, we call getS3Secret by awsaccesskeyid, but during createS3Secret we pass user login name. I think this logic should be modified. # License Header for new classes is wrongly added, it has some GPL header. This needs to be updated. # Can we add end to end robot test to make sure whether this header parsing and validation is happening correctly or not. Already we have tests which configure s3 robot tests.(Where we have configured, some random values, now this can be set using crateS3secret) Or to have a more robust testing, we can have all S3 tests run with secure cluster. I think 2nd approach will be good to have. I am still trying to understand the server side validation, will update if I have any more comments. was (Author: bharatviswa): Hi [~ajayydv] Few comments I have: # Can we use already existing Header parsers AuthorizationHeaderV4 and AuthorizationHeaderV2.java instead of parsing it again in new class AWSV4AuthParser. Same comment for V2 parser. And also can we added reference links, so that it will be easy to refer was header documentation. # And also we have AuthenticationHeaderParser which checks type V2 and V4. And then do required. I think we should do similar checks in OzoneClientProducer.java and then create token? # In OzoneDelegationTokenSecretManager.java, we call getS3Secret by awsaccesskeyid, but during createS3Secret we pass user login name. I think this logic should be modified. # License Header for new classes is wrongly added, it has some GPL header. This needs to be updated. # Can we add end to end robot test to make sure whether this header parsing and validation is happening correctly or not. Already we have tests which configure s3 robot tests.(Where we have configured, some random values, now this can be set using crateS3secret) Or to have a more robust testing, we can have all S3 tests run with secure cluster. I think 2nd approach will be good to have. I am still trying to understand the server side validation, will update if I have any more comments. > Enable token based authentication for S3 api > > > Key: HDDS-1043 > URL: https://issues.apache.org/jira/browse/HDDS-1043 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Labels: security > Fix For: 0.4.0 > > Attachments: HDDS-1043.00.patch, HDDS-1043.01.patch > > > Ozone has a S3 api and mechanism to create S3 like secrets for user. This > jira proposes hadoop compatible token based authentication for S3 api which > utilizes S3 secret stored in OM. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-1043) Enable token based authentication for S3 api
[ https://issues.apache.org/jira/browse/HDDS-1043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16774609#comment-16774609 ] Bharat Viswanadham edited comment on HDDS-1043 at 2/22/19 12:01 AM: Hi [~ajayydv] Few comments I have: # Can we use already existing Header parsers AuthorizationHeaderV4 and AuthorizationHeaderV2.java instead of parsing it again in new class AWSV4AuthParser. Same comment for V2 parser. And also can we added reference links, so that it will be easy to refer was header documentation. # And also we have AuthenticationHeaderParser which checks type V2 and V4. And then do required. I think we should do similar checks in OzoneClientProducer.java and then create token? # In OzoneDelegationTokenSecretManager.java, we call getS3Secret by awsaccesskeyid, but during createS3Secret we pass user login name. I think this logic should be modified. # License Header for new classes is wrongly added, it has some GPL header. This needs to be updated. # Can we add end to end robot test to make sure whether this header parsing and validation is happening correctly or not. Already we have tests which configure s3 robot tests.(Where we have configured, some random values, now this can be set using crateS3secret) Or to have a more robust testing, we can have all S3 tests run with secure cluster. I think 2nd approach will be good to have. I am still trying to understand the server side validation, will update if I have any more comments. was (Author: bharatviswa): Hi [~ajayydv] Few comments I have: # Can we use already existing Header parsers AuthorizationHeaderV4 and AuthorizationHeaderV2.java instead of parsing it again in new class AWSV4AuthParser. Same comment for V2 parser. And also can we added reference links, so that it will be easy to refer was header documentation. # And also we have AuthenticationHeaderParser which checks type V2 and V4. And then do required. I think we should do similar checks in OzoneClientProducer.java and then create token? # In OzoneDelegationTokenSecretManager.java, we call getS3Secret by awsaccesskeyid, but during createS3Secret we pass user login name. I think this logic should be modified. # License Header for new classes is wrongly added, it has some GPL header. This needs to be updated. # Can we add end to end robot test to make sure whether this header parsing and validation is happening correctly or not. Already we have tests which configure s3 robot tests.(Where we have configured, some random values, now this can be set using crateS3secret) Or to have a more robust testing, we can have all S3 tests run with secure cluster. I think 2nd approach will be good to have. I am still trying to understand the server side validation, will update if I have any more comments. > Enable token based authentication for S3 api > > > Key: HDDS-1043 > URL: https://issues.apache.org/jira/browse/HDDS-1043 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Labels: security > Fix For: 0.4.0 > > Attachments: HDDS-1043.00.patch, HDDS-1043.01.patch > > > Ozone has a S3 api and mechanism to create S3 like secrets for user. This > jira proposes hadoop compatible token based authentication for S3 api which > utilizes S3 secret stored in OM. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1043) Enable token based authentication for S3 api
[ https://issues.apache.org/jira/browse/HDDS-1043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16774609#comment-16774609 ] Bharat Viswanadham commented on HDDS-1043: -- Hi [~ajayydv] Few comments I have: # Can we use already existing Header parsers AuthorizationHeaderV4 and AuthorizationHeaderV2.java instead of parsing it again in new class AWSV4AuthParser. Same comment for V2 parser. And also can we added reference links, so that it will be easy to refer was header documentation. # And also we have AuthenticationHeaderParser which checks type V2 and V4. And then do required. I think we should do similar checks in OzoneClientProducer.java and then create token? # In OzoneDelegationTokenSecretManager.java, we call getS3Secret by awsaccesskeyid, but during createS3Secret we pass user login name. I think this logic should be modified. # License Header for new classes is wrongly added, it has some GPL header. This needs to be updated. # Can we add end to end robot test to make sure whether this header parsing and validation is happening correctly or not. Already we have tests which configure s3 robot tests.(Where we have configured, some random values, now this can be set using crateS3secret) Or to have a more robust testing, we can have all S3 tests run with secure cluster. I think 2nd approach will be good to have. I am still trying to understand the server side validation, will update if I have any more comments. > Enable token based authentication for S3 api > > > Key: HDDS-1043 > URL: https://issues.apache.org/jira/browse/HDDS-1043 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Labels: security > Fix For: 0.4.0 > > Attachments: HDDS-1043.00.patch, HDDS-1043.01.patch > > > Ozone has a S3 api and mechanism to create S3 like secrets for user. This > jira proposes hadoop compatible token based authentication for S3 api which > utilizes S3 secret stored in OM. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-1043) Enable token based authentication for S3 api
[ https://issues.apache.org/jira/browse/HDDS-1043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16774609#comment-16774609 ] Bharat Viswanadham edited comment on HDDS-1043 at 2/22/19 12:00 AM: Hi [~ajayydv] Few comments I have: # Can we use already existing Header parsers AuthorizationHeaderV4 and AuthorizationHeaderV2.java instead of parsing it again in new class AWSV4AuthParser. Same comment for V2 parser. And also can we added reference links, so that it will be easy to refer was header documentation. # And also we have AuthenticationHeaderParser which checks type V2 and V4. And then do required. I think we should do similar checks in OzoneClientProducer.java and then create token? # In OzoneDelegationTokenSecretManager.java, we call getS3Secret by awsaccesskeyid, but during createS3Secret we pass user login name. I think this logic should be modified. # License Header for new classes is wrongly added, it has some GPL header. This needs to be updated. # Can we add end to end robot test to make sure whether this header parsing and validation is happening correctly or not. Already we have tests which configure s3 robot tests.(Where we have configured, some random values, now this can be set using crateS3secret) Or to have a more robust testing, we can have all S3 tests run with secure cluster. I think 2nd approach will be good to have. I am still trying to understand the server side validation, will update if I have any more comments. was (Author: bharatviswa): Hi [~ajayydv] Few comments I have: # Can we use already existing Header parsers AuthorizationHeaderV4 and AuthorizationHeaderV2.java instead of parsing it again in new class AWSV4AuthParser. Same comment for V2 parser. And also can we added reference links, so that it will be easy to refer was header documentation. # And also we have AuthenticationHeaderParser which checks type V2 and V4. And then do required. I think we should do similar checks in OzoneClientProducer.java and then create token? # In OzoneDelegationTokenSecretManager.java, we call getS3Secret by awsaccesskeyid, but during createS3Secret we pass user login name. I think this logic should be modified. # License Header for new classes is wrongly added, it has some GPL header. This needs to be updated. # Can we add end to end robot test to make sure whether this header parsing and validation is happening correctly or not. Already we have tests which configure s3 robot tests.(Where we have configured, some random values, now this can be set using crateS3secret) Or to have a more robust testing, we can have all S3 tests run with secure cluster. I think 2nd approach will be good to have. I am still trying to understand the server side validation, will update if I have any more comments. > Enable token based authentication for S3 api > > > Key: HDDS-1043 > URL: https://issues.apache.org/jira/browse/HDDS-1043 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Labels: security > Fix For: 0.4.0 > > Attachments: HDDS-1043.00.patch, HDDS-1043.01.patch > > > Ozone has a S3 api and mechanism to create S3 like secrets for user. This > jira proposes hadoop compatible token based authentication for S3 api which > utilizes S3 secret stored in OM. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14279) [SBN Read] Race condition in ObserverReadProxyProvider
[
https://issues.apache.org/jira/browse/HDFS-14279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16774602#comment-16774602
]
Erik Krogen commented on HDFS-14279:
[~shv], does the new patch look good to you?
> [SBN Read] Race condition in ObserverReadProxyProvider
> --
>
> Key: HDFS-14279
> URL: https://issues.apache.org/jira/browse/HDFS-14279
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs, namenode
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Major
> Attachments: HDFS-14279.000.patch, HDFS-14279.001.patch
>
>
> There is a race condition in {{ObserverReadProxyProvider#getCurrentProxy()}}:
> {code}
> private NNProxyInfo getCurrentProxy() {
> if (currentProxy == null) {
> changeProxy(null);
> }
> return currentProxy;
> }
> {code}
> {{currentProxy}} is a {{volatile}}. Another {{changeProxy()}} could occur
> after the {{changeProxy()}} and before the {{return}}, thus making the return
> value incorrect. I have seen this result in an NPE.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14307) RBF: Update tests to use internal Whitebox instead of Mockito
[ https://issues.apache.org/jira/browse/HDFS-14307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-14307: --- Status: Patch Available (was: Open) > RBF: Update tests to use internal Whitebox instead of Mockito > - > > Key: HDFS-14307 > URL: https://issues.apache.org/jira/browse/HDFS-14307 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: CR Hota >Assignee: CR Hota >Priority: Major > Attachments: HDFS-14307-HDFS-13891.001.patch > > > Router tests should use internal whitebox instead of Mockito. Current router > feature development branch (HDFS-13891) fails without this. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14118) Use DNS to resolve Namenodes and Routers
[ https://issues.apache.org/jira/browse/HDFS-14118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16774581#comment-16774581 ] Fengnan Li commented on HDFS-14118: --- [~yzhangal] I added the answer in the modified hdfs-default.xml. Basically it is for the case that DNS returns a fixed list of namenodes/routers since most of the time the traffic will hit the DNS cache. DNS server itself does return the list in random order (or order in permutation of all ips). By using random it will avoid the client first hit the failed one consistently > Use DNS to resolve Namenodes and Routers > > > Key: HDFS-14118 > URL: https://issues.apache.org/jira/browse/HDFS-14118 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Fengnan Li >Assignee: Fengnan Li >Priority: Major > Attachments: DNS testing log, HDFS design doc_ Single domain name for > clients - Google Docs-1.pdf, HDFS design doc_ Single domain name for clients > - Google Docs.pdf, HDFS-14118.001.patch, HDFS-14118.002.patch, > HDFS-14118.003.patch, HDFS-14118.004.patch, HDFS-14118.005.patch, > HDFS-14118.006.patch, HDFS-14118.007.patch, HDFS-14118.008.patch, > HDFS-14118.009.patch, HDFS-14118.010.patch, HDFS-14118.011.patch, > HDFS-14118.012.patch, HDFS-14118.013.patch, HDFS-14118.014.patch, > HDFS-14118.015.patch, HDFS-14118.016.patch, HDFS-14118.017.patch, > HDFS-14118.018.patch, HDFS-14118.019.patch, HDFS-14118.020.patch, > HDFS-14118.patch > > > Clients will need to know about routers to talk to the HDFS cluster > (obviously), and having routers updating (adding/removing) will have to make > every client change, which is a painful process. > DNS can be used here to resolve the single domain name clients knows to a > list of routers in the current config. However, DNS won't be able to consider > only resolving to the working router based on certain health thresholds. > There are some ways about how this can be solved. One way is to have a > separate script to regularly check the status of the router and update the > DNS records if a router fails the health thresholds. In this way, security > might be carefully considered for this way. Another way is to have the client > do the normal connecting/failover after they get the list of routers, which > requires the change of current failover proxy provider. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1038) Support Service Level Authorization for OM, SCM and DN
[
https://issues.apache.org/jira/browse/HDDS-1038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16774593#comment-16774593
]
Ajay Kumar commented on HDDS-1038:
--
[~xyao] thanks for taking it forward. I am mostly +1 on this. I have a minor
question on javadoc and config name in {{HddsConfigKeys}}. L209-215: Protocol
name is SCMSecurityProtocol. Is naming of corresponding config intentional or a
typo?
> Support Service Level Authorization for OM, SCM and DN
> --
>
> Key: HDDS-1038
> URL: https://issues.apache.org/jira/browse/HDDS-1038
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Ajay Kumar
>Assignee: Ajay Kumar
>Priority: Major
> Labels: Security
> Fix For: 0.4.0
>
> Attachments: HDDS-1038.00.patch, HDDS-1038.01.patch,
> HDDS-1038.02.patch, HDDS-1038.03.patch, HDDS-1038.04.patch,
> HDDS-1038.05.patch, HDDS-1038.06.patch, HDDS-1038.07.patch
>
>
> In a secure Ozone cluster. Datanodes fail to connect to SCM on
> {{StorageContainerDatanodeProtocol}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1061) DelegationToken: Add certificate serial id to Ozone Delegation Token Identifier
[ https://issues.apache.org/jira/browse/HDDS-1061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16774584#comment-16774584 ] Ajay Kumar commented on HDDS-1061: -- [~xyao] After getting certificate for other OM instance we need few more capabilities like: # How to persist certificate for other om instance? (either in trust store or parallel to default certificate file ) ## If truststore than where and how to store its secret key. # load it when OM starts. Since all of this is part of HA, is it ok if we handle it separately in [HDDS-1118]? > DelegationToken: Add certificate serial id to Ozone Delegation Token > Identifier > > > Key: HDDS-1061 > URL: https://issues.apache.org/jira/browse/HDDS-1061 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Attachments: HDDS-1061.00.patch, HDDS-1061.01.patch, > HDDS-1061.02.patch, HDDS-1061.03.patch > > > 1. Add certificate serial id to Ozone Delegation Token Identifier. Required > for OM HA support. > 2. Validate Ozone token based on public key from OM certificate -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (HDDS-1038) Support Service Level Authorization for OM, SCM and DN
[
https://issues.apache.org/jira/browse/HDDS-1038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ajay Kumar updated HDDS-1038:
-
Comment: was deleted
(was: [~xyao] After getting certificate for other OM instance we need few more
capabilities like:
# How to persist certificate for other om instance? (either in trust store or
parallel to default certificate file )
## If truststore than where and how to store its secret key.
# load it when OM starts.
Since all of this is part of HA, is it ok if we handle it separately in
[HDDS-1118]?)
> Support Service Level Authorization for OM, SCM and DN
> --
>
> Key: HDDS-1038
> URL: https://issues.apache.org/jira/browse/HDDS-1038
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Ajay Kumar
>Assignee: Ajay Kumar
>Priority: Major
> Labels: Security
> Fix For: 0.4.0
>
> Attachments: HDDS-1038.00.patch, HDDS-1038.01.patch,
> HDDS-1038.02.patch, HDDS-1038.03.patch, HDDS-1038.04.patch,
> HDDS-1038.05.patch, HDDS-1038.06.patch, HDDS-1038.07.patch
>
>
> In a secure Ozone cluster. Datanodes fail to connect to SCM on
> {{StorageContainerDatanodeProtocol}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1038) Support Service Level Authorization for OM, SCM and DN
[
https://issues.apache.org/jira/browse/HDDS-1038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16774583#comment-16774583
]
Ajay Kumar commented on HDDS-1038:
--
[~xyao] After getting certificate for other OM instance we need few more
capabilities like:
# How to persist certificate for other om instance? (either in trust store or
parallel to default certificate file )
## If truststore than where and how to store its secret key.
# load it when OM starts.
Since all of this is part of HA, is it ok if we handle it separately in
[HDDS-1118]?
> Support Service Level Authorization for OM, SCM and DN
> --
>
> Key: HDDS-1038
> URL: https://issues.apache.org/jira/browse/HDDS-1038
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Ajay Kumar
>Assignee: Ajay Kumar
>Priority: Major
> Labels: Security
> Fix For: 0.4.0
>
> Attachments: HDDS-1038.00.patch, HDDS-1038.01.patch,
> HDDS-1038.02.patch, HDDS-1038.03.patch, HDDS-1038.04.patch,
> HDDS-1038.05.patch, HDDS-1038.06.patch, HDDS-1038.07.patch
>
>
> In a secure Ozone cluster. Datanodes fail to connect to SCM on
> {{StorageContainerDatanodeProtocol}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14308) DFSStripedInputStream should implement unbuffer()
[
https://issues.apache.org/jira/browse/HDFS-14308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joe McDonnell updated HDFS-14308:
-
Attachment: ec_heap_dump.png
> DFSStripedInputStream should implement unbuffer()
> -
>
> Key: HDFS-14308
> URL: https://issues.apache.org/jira/browse/HDFS-14308
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Joe McDonnell
>Priority: Major
> Attachments: ec_heap_dump.png
>
>
> Some users of HDFS cache opened HDFS file handles to avoid repeated
> roundtrips to the NameNode. For example, Impala caches up to 20,000 HDFS file
> handles by default. Recent tests on erasure coded files show that the open
> file handles can consume a large amount of memory when not in use.
> For example, here is output from Impala's JMX endpoint when 608 file handles
> are cached
> {noformat}
> {
> "name": "java.nio:type=BufferPool,name=direct",
> "modelerType": "sun.management.ManagementFactoryHelper$1",
> "Name": "direct",
> "TotalCapacity": 1921048960,
> "MemoryUsed": 1921048961,
> "Count": 633,
> "ObjectName": "java.nio:type=BufferPool,name=direct"
> },{noformat}
> This shows direct buffer memory usage of 3MB per DFSStripedInputStream.
> Attached is output from Eclipse MAT showing that the direct buffers come from
> DFSStripedInputStream objects.
> To support caching file handles on erasure coded files, DFSStripedInputStream
> should implement the unbuffer() call. See HDFS-7694. "unbuffer()" is intended
> to move an input stream to a lower memory state to support these caching use
> cases. Both Impala and HBase call unbuffer() when a file handle is being
> cached and potentially unused for significant chunks of time.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14308) DFSStripedInputStream should implement unbuffer()
[
https://issues.apache.org/jira/browse/HDFS-14308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16774576#comment-16774576
]
Joe McDonnell commented on HDFS-14308:
--
Adding a link to the original IMPALA issue
> DFSStripedInputStream should implement unbuffer()
> -
>
> Key: HDFS-14308
> URL: https://issues.apache.org/jira/browse/HDFS-14308
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Joe McDonnell
>Priority: Major
> Attachments: ec_heap_dump.png
>
>
> Some users of HDFS cache opened HDFS file handles to avoid repeated
> roundtrips to the NameNode. For example, Impala caches up to 20,000 HDFS file
> handles by default. Recent tests on erasure coded files show that the open
> file handles can consume a large amount of memory when not in use.
> For example, here is output from Impala's JMX endpoint when 608 file handles
> are cached
> {noformat}
> {
> "name": "java.nio:type=BufferPool,name=direct",
> "modelerType": "sun.management.ManagementFactoryHelper$1",
> "Name": "direct",
> "TotalCapacity": 1921048960,
> "MemoryUsed": 1921048961,
> "Count": 633,
> "ObjectName": "java.nio:type=BufferPool,name=direct"
> },{noformat}
> This shows direct buffer memory usage of 3MB per DFSStripedInputStream.
> Attached is output from Eclipse MAT showing that the direct buffers come from
> DFSStripedInputStream objects.
> To support caching file handles on erasure coded files, DFSStripedInputStream
> should implement the unbuffer() call. See HDFS-7694. "unbuffer()" is intended
> to move an input stream to a lower memory state to support these caching use
> cases. Both Impala and HBase call unbuffer() when a file handle is being
> cached and potentially unused for significant chunks of time.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-14308) DFSStripedInputStream should implement unbuffer()
Joe McDonnell created HDFS-14308:
Summary: DFSStripedInputStream should implement unbuffer()
Key: HDFS-14308
URL: https://issues.apache.org/jira/browse/HDFS-14308
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Joe McDonnell
Attachments: ec_heap_dump.png
Some users of HDFS cache opened HDFS file handles to avoid repeated roundtrips
to the NameNode. For example, Impala caches up to 20,000 HDFS file handles by
default. Recent tests on erasure coded files show that the open file handles
can consume a large amount of memory when not in use.
For example, here is output from Impala's JMX endpoint when 608 file handles
are cached
{noformat}
{
"name": "java.nio:type=BufferPool,name=direct",
"modelerType": "sun.management.ManagementFactoryHelper$1",
"Name": "direct",
"TotalCapacity": 1921048960,
"MemoryUsed": 1921048961,
"Count": 633,
"ObjectName": "java.nio:type=BufferPool,name=direct"
},{noformat}
This shows direct buffer memory usage of 3MB per DFSStripedInputStream.
Attached is output from Eclipse MAT showing that the direct buffers come from
DFSStripedInputStream objects.
To support caching file handles on erasure coded files, DFSStripedInputStream
should implement the unbuffer() call. See HDFS-7694. "unbuffer()" is intended
to move an input stream to a lower memory state to support these caching use
cases. Both Impala and HBase call unbuffer() when a file handle is being cached
and potentially unused for significant chunks of time.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14292) Introduce Java ExecutorService to DataXceiverServer
[
https://issues.apache.org/jira/browse/HDFS-14292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16774570#comment-16774570
]
Hadoop QA commented on HDFS-14292:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
42s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 26s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
18s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m
22s{color} | {color:green} hadoop-hdfs-project generated 0 new + 538 unchanged
- 2 fixed = 538 total (was 540) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 9s{color} | {color:orange} hadoop-hdfs-project: The patch generated 3 new +
632 unchanged - 5 fixed = 635 total (was 637) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 37s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
59s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}129m 19s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
51s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}203m 58s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.TestFsck |
| | hadoop.hdfs.server.balancer.TestBalancer |
| | hadoop.hdfs.server.datanode.TestBlockRecovery |
| | hadoop.hdfs.qjournal.server.TestJournalNodeSync |
| | hadoop.hdfs.server.datanode.TestBPOfferService |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-14292 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12959661/HDFS-14292.6.patch |
| Optional Tests | dupname asflicense compile javac
[jira] [Commented] (HDFS-14052) RBF: Use Router keytab for WebHDFS
[
https://issues.apache.org/jira/browse/HDFS-14052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16774575#comment-16774575
]
CR Hota commented on HDFS-14052:
[~elgoiri] Thanks for the update
Created HDFS-14307 to update the test.
> RBF: Use Router keytab for WebHDFS
> --
>
> Key: HDFS-14052
> URL: https://issues.apache.org/jira/browse/HDFS-14052
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: CR Hota
>Priority: Major
> Attachments: HDFS-14052-HDFS-13891.0.patch,
> HDFS-14052-HDFS-13891.1.patch
>
>
> When the RouterHttpServer starts it does:
> {code}
> NameNodeHttpServer.initWebHdfs(conf, httpAddress.getHostName(),
> httpServer,
> RouterWebHdfsMethods.class.getPackage().getName());
> {code}
> This function is in the NN and is pretty generic.
> However, it then calls to NameNodeHttpServer#getAuthFilterParams, which does:
> {code}
> String httpKeytab = conf.get(DFSUtil.getSpnegoKeytabKey(conf,
> DFSConfigKeys.DFS_NAMENODE_KEYTAB_FILE_KEY));
> {code}
> In most cases, the regular web keytab will kick in, but we should make this a
> parameter and load the Router one just in case.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14307) RBF: Update tests to use internal Whitebox instead of Mockito
[ https://issues.apache.org/jira/browse/HDFS-14307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] CR Hota updated HDFS-14307: --- Attachment: HDFS-14307-HDFS-13891.001.patch > RBF: Update tests to use internal Whitebox instead of Mockito > - > > Key: HDFS-14307 > URL: https://issues.apache.org/jira/browse/HDFS-14307 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: CR Hota >Assignee: CR Hota >Priority: Major > Attachments: HDFS-14307-HDFS-13891.001.patch > > > Router tests should use internal whitebox instead of Mockito. Current router > feature development branch (HDFS-13891) fails without this. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14307) RBF: Update tests to use internal Whitebox instead of Mockito
[ https://issues.apache.org/jira/browse/HDFS-14307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] CR Hota updated HDFS-14307: --- Attachment: (was: HDFS-4307-HDFS-13891.001.patch) > RBF: Update tests to use internal Whitebox instead of Mockito > - > > Key: HDFS-14307 > URL: https://issues.apache.org/jira/browse/HDFS-14307 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: CR Hota >Assignee: CR Hota >Priority: Major > > Router tests should use internal whitebox instead of Mockito. Current router > feature development branch (HDFS-13891) fails without this. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14307) RBF: Update tests to use internal Whitebox instead of Mockito
[ https://issues.apache.org/jira/browse/HDFS-14307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] CR Hota updated HDFS-14307: --- Attachment: HDFS-4307-HDFS-13891.001.patch > RBF: Update tests to use internal Whitebox instead of Mockito > - > > Key: HDFS-14307 > URL: https://issues.apache.org/jira/browse/HDFS-14307 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: CR Hota >Assignee: CR Hota >Priority: Major > > Router tests should use internal whitebox instead of Mockito. Current router > feature development branch (HDFS-13891) fails without this. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14307) RBF: Update tests to use internal Whitebox instead of MockitoRBF
[ https://issues.apache.org/jira/browse/HDFS-14307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] CR Hota updated HDFS-14307: --- Summary: RBF: Update tests to use internal Whitebox instead of MockitoRBF (was: RBF: Update tests to use internal Whitebox instead of MockitoRBF:) > RBF: Update tests to use internal Whitebox instead of MockitoRBF > > > Key: HDFS-14307 > URL: https://issues.apache.org/jira/browse/HDFS-14307 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: CR Hota >Assignee: CR Hota >Priority: Major > > Router tests should use internal whitebox instead of Mockito. Current router > feature development branch (HDFS-13891) fails without this. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14118) Use DNS to resolve Namenodes and Routers
[ https://issues.apache.org/jira/browse/HDFS-14118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16774569#comment-16774569 ] Yongjun Zhang commented on HDFS-14118: -- Thanks [~fengnanli], would you please address the questions I asked in #4? thanks. > Use DNS to resolve Namenodes and Routers > > > Key: HDFS-14118 > URL: https://issues.apache.org/jira/browse/HDFS-14118 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Fengnan Li >Assignee: Fengnan Li >Priority: Major > Attachments: DNS testing log, HDFS design doc_ Single domain name for > clients - Google Docs-1.pdf, HDFS design doc_ Single domain name for clients > - Google Docs.pdf, HDFS-14118.001.patch, HDFS-14118.002.patch, > HDFS-14118.003.patch, HDFS-14118.004.patch, HDFS-14118.005.patch, > HDFS-14118.006.patch, HDFS-14118.007.patch, HDFS-14118.008.patch, > HDFS-14118.009.patch, HDFS-14118.010.patch, HDFS-14118.011.patch, > HDFS-14118.012.patch, HDFS-14118.013.patch, HDFS-14118.014.patch, > HDFS-14118.015.patch, HDFS-14118.016.patch, HDFS-14118.017.patch, > HDFS-14118.018.patch, HDFS-14118.019.patch, HDFS-14118.020.patch, > HDFS-14118.patch > > > Clients will need to know about routers to talk to the HDFS cluster > (obviously), and having routers updating (adding/removing) will have to make > every client change, which is a painful process. > DNS can be used here to resolve the single domain name clients knows to a > list of routers in the current config. However, DNS won't be able to consider > only resolving to the working router based on certain health thresholds. > There are some ways about how this can be solved. One way is to have a > separate script to regularly check the status of the router and update the > DNS records if a router fails the health thresholds. In this way, security > might be carefully considered for this way. Another way is to have the client > do the normal connecting/failover after they get the list of routers, which > requires the change of current failover proxy provider. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14307) RBF: Update tests to use internal Whitebox instead of Mockito
[ https://issues.apache.org/jira/browse/HDFS-14307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] CR Hota updated HDFS-14307: --- Summary: RBF: Update tests to use internal Whitebox instead of Mockito (was: RBF: Update tests to use internal Whitebox instead of MockitoRBF) > RBF: Update tests to use internal Whitebox instead of Mockito > - > > Key: HDFS-14307 > URL: https://issues.apache.org/jira/browse/HDFS-14307 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: CR Hota >Assignee: CR Hota >Priority: Major > > Router tests should use internal whitebox instead of Mockito. Current router > feature development branch (HDFS-13891) fails without this. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14307) RBF: Update tests to use internal Whitebox instead of MockitoRBF:
[ https://issues.apache.org/jira/browse/HDFS-14307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] CR Hota updated HDFS-14307: --- Summary: RBF: Update tests to use internal Whitebox instead of MockitoRBF: (was: Update tests to use internal Whitebox instead of Mockito) > RBF: Update tests to use internal Whitebox instead of MockitoRBF: > - > > Key: HDFS-14307 > URL: https://issues.apache.org/jira/browse/HDFS-14307 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: CR Hota >Assignee: CR Hota >Priority: Major > > Router tests should use internal whitebox instead of Mockito. Current router > feature development branch (HDFS-13891) fails without this. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-14307) Update tests to use internal Whitebox instead of Mockito
CR Hota created HDFS-14307: -- Summary: Update tests to use internal Whitebox instead of Mockito Key: HDFS-14307 URL: https://issues.apache.org/jira/browse/HDFS-14307 Project: Hadoop HDFS Issue Type: Improvement Reporter: CR Hota Assignee: CR Hota Router tests should use internal whitebox instead of Mockito. Current router feature development branch (HDFS-13891) fails without this. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14052) RBF: Use Router keytab for WebHDFS
[
https://issues.apache.org/jira/browse/HDFS-14052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16774559#comment-16774559
]
Íñigo Goiri commented on HDFS-14052:
I think I introduced the wrong use in HDFS-14082 and then with the rebase to
trunk, this got deprecated.
[~crh] do you mind opening a JIRA fixing the use from:
{code}
org.mockito.internal.util.reflection.Whitebox;
{code}
to:
{code}
org.apache.hadoop.test.Whitebox;
{code}
?
> RBF: Use Router keytab for WebHDFS
> --
>
> Key: HDFS-14052
> URL: https://issues.apache.org/jira/browse/HDFS-14052
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: CR Hota
>Priority: Major
> Attachments: HDFS-14052-HDFS-13891.0.patch,
> HDFS-14052-HDFS-13891.1.patch
>
>
> When the RouterHttpServer starts it does:
> {code}
> NameNodeHttpServer.initWebHdfs(conf, httpAddress.getHostName(),
> httpServer,
> RouterWebHdfsMethods.class.getPackage().getName());
> {code}
> This function is in the NN and is pretty generic.
> However, it then calls to NameNodeHttpServer#getAuthFilterParams, which does:
> {code}
> String httpKeytab = conf.get(DFSUtil.getSpnegoKeytabKey(conf,
> DFSConfigKeys.DFS_NAMENODE_KEYTAB_FILE_KEY));
> {code}
> In most cases, the regular web keytab will kick in, but we should make this a
> parameter and load the Router one just in case.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
