[jira] [Created] (HDFS-12897) Path not found when we get the ec policy for a .snapshot dir
Harshakiran Reddy created HDFS-12897: Summary: Path not found when we get the ec policy for a .snapshot dir Key: HDFS-12897 URL: https://issues.apache.org/jira/browse/HDFS-12897 Project: Hadoop HDFS Issue Type: Bug Components: erasure-coding, hdfs, snapshots Affects Versions: 3.0.0-alpha1 Reporter: Harshakiran Reddy Scenario:- --- bin> ./hdfs ec -getPolicy -path /dir/ 2017-12-06 12:51:35,324 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable RS-3-2-1024k bin> ./hdfs ec -getPolicy -path /dir/.snapshot/ 2017-12-06 12:19:52,204 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable {{FileNotFoundException: Path not found: /dir/.snapshot}} bin> ./hdfs dfs -ls /dir/.snapshot/ 2017-12-06 12:20:12,956 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Found 2 items drwxr-xr-x - user group 0 2017-12-05 12:27 /dir/.snapshot/s1 drwxr-xr-x - user group 0 2017-12-05 12:28 /dir/.snapshot/s2 bin> ./hdfs storagepolicies -getStoragePolicy -path /dir/.snapshot/ 2017-12-06 12:24:10,729 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable {{The storage policy of /dir/.snapshot/ is unspecified}} bin> ./hdfs storagepolicies -getStoragePolicy -path /dir/ 2017-12-06 12:50:17,010 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable The storage policy of /dir/: BlockStoragePolicy{COLD:2, storageTypes=[ARCHIVE], creationFallbacks=[], replicationFallbacks=[]} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-12896) when set replicate EC policy for a directory or file,it's EC policy cannot be querying by getPolicy command.
chencan created HDFS-12896: -- Summary: when set replicate EC policy for a directory or file,it's EC policy cannot be querying by getPolicy command. Key: HDFS-12896 URL: https://issues.apache.org/jira/browse/HDFS-12896 Project: Hadoop HDFS Issue Type: Bug Reporter: chencan When i set replicate EC policy for ecDir,then query it by getPolicy,it return ‘The erasure coding policy of /ecDir is unspecified', as follow. [root@master bin]# hdfs dfs -mkdir /ecDir [root@master bin]# hdfs ec -setPolicy -path /ecDir -replicate Set erasure coding policy replication on /ecDir [root@master bin]# hdfs ec -getPolicy -path /ecDir The erasure coding policy of /ecDir is unspecified -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-12895) RBF: Add ACL support for mount table
Yiqun Lin created HDFS-12895: Summary: RBF: Add ACL support for mount table Key: HDFS-12895 URL: https://issues.apache.org/jira/browse/HDFS-12895 Project: Hadoop HDFS Issue Type: Sub-task Affects Versions: 3.0.0-alpha3 Reporter: Yiqun Lin Assignee: Yiqun Lin -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: VOTE] Release Apache Hadoop 2.7.5 (RC0)
+1 (non-binding) Thanks for driving this. --Built from the source. --Installed 3 Node Cluster. --Ran basic shell commands. --Ran sample jobs, pi,wordcount. Thanks, Ajay On 12/1/17, 8:42 PM, "Konstantin Shvachko" wrote: Hi everybody, This is the next dot release of Apache Hadoop 2.7 line. The previous one 2.7.4 was release August 4, 2017. Release 2.7.5 includes critical bug fixes and optimizations. See more details in Release Note: http://home.apache.org/~shv/hadoop-2.7.5-RC0/releasenotes.html The RC0 is available at: http://home.apache.org/~shv/hadoop-2.7.5-RC0/ Please give it a try and vote on this thread. The vote will run for 5 days ending 12/08/2017. My up to date public key is available from: https://dist.apache.org/repos/dist/release/hadoop/common/KEYS Thanks, --Konstantin
[jira] [Created] (HDFS-12894) [READ] Skip setting block count of ProvidedStorageInfo on DN registration update
Virajith Jalaparti created HDFS-12894: - Summary: [READ] Skip setting block count of ProvidedStorageInfo on DN registration update Key: HDFS-12894 URL: https://issues.apache.org/jira/browse/HDFS-12894 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Virajith Jalaparti Assignee: Virajith Jalaparti -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-12893) [READ] Support replication of Provided blocks with non-default topologies.
Virajith Jalaparti created HDFS-12893: - Summary: [READ] Support replication of Provided blocks with non-default topologies. Key: HDFS-12893 URL: https://issues.apache.org/jira/browse/HDFS-12893 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Virajith Jalaparti -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: [DISCUSS] Merge HDFS-9806 to trunk
I'm generally supportive of a merge - unless anyone brings up new concerns, I've been happy with the design, implementation and discussion and it seems to be at a good checkpoint. When you get to writing the site documentation - one thing in particular I think should be called out early in the description of the feature is the need to create a read-only namespace specific to the remote storage. I can see this getting merged and more than a few people wanting to know how they can dynamically mount their remote storage on their existing HDFS cluster today :) On Sun, Dec 3, 2017 at 3:43 PM, Chris Douglas wrote: > Hey hdfs-dev@- > > The HDFS-9806 dev branch is getting ready to merge to trunk. This branch > adds a new storage type (PROVIDED) to support reading remote data as HDFS > blocks. Design documentation and discussion are available on JIRA [1]. > > Over the next week or so, we'll work through the remaining issues, write > the site documentation, and generally clean up the branch. The current code > has been stressed in experimental settings, including a large production > benchmark. When this feature is not enabled and active, it should have no > effect on HDFS. The merge will checkpoint progress on reading remote data > through HDFS, setting up work on the write path [2] and improving > usability. > > Please look over the branch and raise issues that should be addressed > before the merge. -C > > [1]: https://issues.apache.org/jira/browse/HDFS-9806 > [2]: https://issues.apache.org/jira/browse/HDFS-12090 >
[jira] [Reopened] (HDFS-12891) TestClusterTopology#testChooseRandom fails intermittently
[ https://issues.apache.org/jira/browse/HDFS-12891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zsolt Venczel reopened HDFS-12891: -- > TestClusterTopology#testChooseRandom fails intermittently > - > > Key: HDFS-12891 > URL: https://issues.apache.org/jira/browse/HDFS-12891 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.9.0, 3.0.0-beta1, 2.8.2 >Reporter: Zsolt Venczel >Assignee: Zsolt Venczel > Labels: flaky-test > Fix For: 2.9.0, 3.0.0-beta1, 2.8.3, 3.1.0 > > > {noformat} > java.net.BindException: Problem binding to [localhost:36701] > java.net.BindException: Address already in use; For more details see: > http://wiki.apache.org/hadoop/BindException > at sun.nio.ch.Net.bind0(Native Method) > at sun.nio.ch.Net.bind(Net.java:433) > at sun.nio.ch.Net.bind(Net.java:425) > at > sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223) > at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) > at org.apache.hadoop.ipc.Server.bind(Server.java:546) > at org.apache.hadoop.ipc.Server$Listener.(Server.java:955) > at org.apache.hadoop.ipc.Server.(Server.java:2655) > at org.apache.hadoop.ipc.RPC$Server.(RPC.java:968) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server.(ProtobufRpcEngine.java:367) > at > org.apache.hadoop.ipc.ProtobufRpcEngine.getServer(ProtobufRpcEngine.java:342) > at org.apache.hadoop.ipc.RPC$Builder.build(RPC.java:810) > at > org.apache.hadoop.hdfs.server.datanode.DataNode.initIpcServer(DataNode.java:954) > at > org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:1314) > at > org.apache.hadoop.hdfs.server.datanode.DataNode.(DataNode.java:481) > at > org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2611) > at > org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2499) > at > org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2546) > at > org.apache.hadoop.hdfs.MiniDFSCluster.restartDataNode(MiniDFSCluster.java:2152) > at > org.apache.hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock.testPendingDeleteUnknownBlocks(TestPendingInvalidateBlock.java:175) > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-12892) TestClusterTopology#testChooseRandom fails intermittently
Zsolt Venczel created HDFS-12892: Summary: TestClusterTopology#testChooseRandom fails intermittently Key: HDFS-12892 URL: https://issues.apache.org/jira/browse/HDFS-12892 Project: Hadoop HDFS Issue Type: Bug Components: test Affects Versions: 3.0.0 Reporter: Zsolt Venczel Flaky test failure: {code:java} java.lang.AssertionError Error Not choosing nodes randomly Stack Trace java.lang.AssertionError: Not choosing nodes randomly at org.apache.hadoop.net.TestClusterTopology.testChooseRandom(TestClusterTopology.java:170) {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-12891) TestClusterTopology#testChooseRandom fails intermittently
[ https://issues.apache.org/jira/browse/HDFS-12891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zsolt Venczel resolved HDFS-12891. -- Resolution: Invalid > TestClusterTopology#testChooseRandom fails intermittently > - > > Key: HDFS-12891 > URL: https://issues.apache.org/jira/browse/HDFS-12891 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.9.0, 3.0.0-beta1, 2.8.2 >Reporter: Zsolt Venczel >Assignee: Eric Badger > Labels: flaky-test > Fix For: 2.8.3, 3.1.0, 3.0.0-beta1, 2.9.0 > > > {noformat} > java.net.BindException: Problem binding to [localhost:36701] > java.net.BindException: Address already in use; For more details see: > http://wiki.apache.org/hadoop/BindException > at sun.nio.ch.Net.bind0(Native Method) > at sun.nio.ch.Net.bind(Net.java:433) > at sun.nio.ch.Net.bind(Net.java:425) > at > sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223) > at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) > at org.apache.hadoop.ipc.Server.bind(Server.java:546) > at org.apache.hadoop.ipc.Server$Listener.(Server.java:955) > at org.apache.hadoop.ipc.Server.(Server.java:2655) > at org.apache.hadoop.ipc.RPC$Server.(RPC.java:968) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server.(ProtobufRpcEngine.java:367) > at > org.apache.hadoop.ipc.ProtobufRpcEngine.getServer(ProtobufRpcEngine.java:342) > at org.apache.hadoop.ipc.RPC$Builder.build(RPC.java:810) > at > org.apache.hadoop.hdfs.server.datanode.DataNode.initIpcServer(DataNode.java:954) > at > org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:1314) > at > org.apache.hadoop.hdfs.server.datanode.DataNode.(DataNode.java:481) > at > org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2611) > at > org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2499) > at > org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2546) > at > org.apache.hadoop.hdfs.MiniDFSCluster.restartDataNode(MiniDFSCluster.java:2152) > at > org.apache.hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock.testPendingDeleteUnknownBlocks(TestPendingInvalidateBlock.java:175) > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-12891) TestClusterTopology#testChooseRandom fails intermittently
Zsolt Venczel created HDFS-12891: Summary: TestClusterTopology#testChooseRandom fails intermittently Key: HDFS-12891 URL: https://issues.apache.org/jira/browse/HDFS-12891 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.9.0, 3.0.0-beta1, 2.8.2 Reporter: Zsolt Venczel Assignee: Eric Badger Fix For: 2.9.0, 3.0.0-beta1, 2.8.3, 3.1.0 {noformat} java.net.BindException: Problem binding to [localhost:36701] java.net.BindException: Address already in use; For more details see: http://wiki.apache.org/hadoop/BindException at sun.nio.ch.Net.bind0(Native Method) at sun.nio.ch.Net.bind(Net.java:433) at sun.nio.ch.Net.bind(Net.java:425) at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223) at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) at org.apache.hadoop.ipc.Server.bind(Server.java:546) at org.apache.hadoop.ipc.Server$Listener.(Server.java:955) at org.apache.hadoop.ipc.Server.(Server.java:2655) at org.apache.hadoop.ipc.RPC$Server.(RPC.java:968) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server.(ProtobufRpcEngine.java:367) at org.apache.hadoop.ipc.ProtobufRpcEngine.getServer(ProtobufRpcEngine.java:342) at org.apache.hadoop.ipc.RPC$Builder.build(RPC.java:810) at org.apache.hadoop.hdfs.server.datanode.DataNode.initIpcServer(DataNode.java:954) at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:1314) at org.apache.hadoop.hdfs.server.datanode.DataNode.(DataNode.java:481) at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2611) at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2499) at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2546) at org.apache.hadoop.hdfs.MiniDFSCluster.restartDataNode(MiniDFSCluster.java:2152) at org.apache.hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock.testPendingDeleteUnknownBlocks(TestPendingInvalidateBlock.java:175) {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[VOTE] Release Apache Hadoop 2.8.3 (RC0)
Hi all, I've created the first release candidate (RC0) for Apache Hadoop 2.8.3. This is our next maint release to follow up 2.8.2. It includes 79 important fixes and improvements. The RC artifacts are available at: http://home.apache.org/~junping_du/hadoop-2.8.3-RC0 The RC tag in git is: release-2.8.3-RC0 The maven artifacts are available via repository.apache.org at: https://repository.apache.org/content/repositories/orgapachehadoop-1072 Please try the release and vote; the vote will run for the usual 5 working days, ending on 12/12/2017 PST time. Thanks, Junping
Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/60/ No changes -1 overall The following subsystems voted -1: asflicense unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Unreaped Processes : hadoop-common:1 hadoop-hdfs:16 bkjournal:5 hadoop-yarn-server-nodemanager:1 hadoop-yarn-server-timelineservice:1 hadoop-yarn-client:6 hadoop-yarn-applications-distributedshell:1 hadoop-mapreduce-client-app:2 hadoop-mapreduce-client-jobclient:12 hadoop-streaming:1 hadoop-distcp:4 Failed junit tests : hadoop.http.TestHttpServer hadoop.hdfs.web.TestWebHdfsWithAuthenticationFilter hadoop.hdfs.web.TestWebHdfsWithMultipleNameNodes hadoop.hdfs.web.TestFSMainOperationsWebHdfs hadoop.yarn.server.nodemanager.webapp.TestNMWebServer hadoop.yarn.server.resourcemanager.reservation.TestCapacityOverTimePolicy hadoop.yarn.applications.distributedshell.TestDistributedShellWithNodeLabels hadoop.mapred.TestReduceFetch hadoop.mapreduce.security.TestBinaryTokenFile hadoop.mapreduce.security.TestJHSSecurity hadoop.fs.TestFileSystem hadoop.mapreduce.security.TestMRCredentials hadoop.conf.TestNoDefaultsJobConf hadoop.fs.TestDFSIO hadoop.mapred.TestJobSysDirWithDFS hadoop.tools.TestIntegration hadoop.tools.TestDistCpViewFs hadoop.resourceestimator.solver.impl.TestLpSolver hadoop.resourceestimator.service.TestResourceEstimatorService Timed out junit tests : org.apache.hadoop.log.TestLogLevel org.apache.hadoop.hdfs.TestLeaseRecovery2 org.apache.hadoop.hdfs.TestRead org.apache.hadoop.security.TestPermission org.apache.hadoop.hdfs.TestDFSInotifyEventInputStream org.apache.hadoop.hdfs.TestDatanodeLayoutUpgrade org.apache.hadoop.hdfs.TestFileAppendRestart org.apache.hadoop.hdfs.web.TestWebHdfsWithRestCsrfPreventionFilter org.apache.hadoop.hdfs.TestDFSMkdirs org.apache.hadoop.hdfs.TestDFSOutputStream org.apache.hadoop.hdfs.web.TestWebHDFS org.apache.hadoop.hdfs.web.TestWebHDFSXAttr org.apache.hadoop.metrics2.sink.TestRollingFileSystemSinkWithHdfs org.apache.hadoop.hdfs.TestDistributedFileSystem org.apache.hadoop.hdfs.TestReplaceDatanodeFailureReplication org.apache.hadoop.hdfs.TestDFSShell org.apache.hadoop.contrib.bkjournal.TestBootstrapStandbyWithBKJM org.apache.hadoop.contrib.bkjournal.TestBookKeeperJournalManager org.apache.hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints org.apache.hadoop.contrib.bkjournal.TestBookKeeperAsHASharedDir org.apache.hadoop.contrib.bkjournal.TestBookKeeperSpeculativeRead org.apache.hadoop.yarn.server.nodemanager.TestNodeStatusUpdater org.apache.hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServices org.apache.hadoop.yarn.client.TestRMFailover org.apache.hadoop.yarn.client.cli.TestYarnCLI org.apache.hadoop.yarn.client.TestApplicationClientProtocolOnHA org.apache.hadoop.yarn.client.api.impl.TestYarnClientWithReservation org.apache.hadoop.yarn.client.api.impl.TestYarnClient org.apache.hadoop.yarn.client.api.impl.TestAMRMClient org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell org.apache.hadoop.mapreduce.v2.app.TestStagingCleanup org.apache.hadoop.mapreduce.v2.app.TestJobEndNotifier org.apache.hadoop.mapred.lib.TestDelegatingInputFormat org.apache.hadoop.mapred.TestMiniMRClasspath org.apache.hadoop.mapred.TestClusterMapReduceTestCase org.apache.hadoop.mapreduce.TestMapCollection org.apache.hadoop.mapred.TestMRIntermediateDataEncryption org.apache.hadoop.mapred.TestMRTimelineEventHandling org.apache.hadoop.mapred.join.TestDatamerge org.apache.hadoop.mapreduce.security.ssl.TestEncryptedShuffle org.apache.hadoop.mapreduce.TestMapReduce org.apache.hadoop.mapred.TestReduceFetchFromPartialMem org.apache.hadoop.mapred.TestLazyOutput org.apache.hadoop.streaming.TestMultipleCachefiles org.apache.hadoop.tools.TestDistCpSync org.apache.hadoop.tools.TestDistCpWithXAttrs org.apache.hadoop.tools.TestDistCpSyncReverseFromTarget org.apache.hadoop.tools.TestDistCpSyncReverseFromSource cc: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/60/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-branch2-
[jira] [Created] (HDFS-12890) XceiverClient should have upper bound on async requests
Shashikant Banerjee created HDFS-12890: -- Summary: XceiverClient should have upper bound on async requests Key: HDFS-12890 URL: https://issues.apache.org/jira/browse/HDFS-12890 Project: Hadoop HDFS Issue Type: Sub-task Components: HDFS-7240 Affects Versions: HDFS-7240 Reporter: Shashikant Banerjee Assignee: Shashikant Banerjee Fix For: HDFS-7240 XceiverClient-ratis maintains upper bound on the no of outstanding async requests . XceiverClient should also impose an upper bound on the no of outstanding async requests received from client for write. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org