[jira] [Reopened] (HDDS-1659) Define the process to add proposal/design docs to the Ozone subproject
[ https://issues.apache.org/jira/browse/HDDS-1659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton reopened HDDS-1659: > Define the process to add proposal/design docs to the Ozone subproject > -- > > Key: HDDS-1659 > URL: https://issues.apache.org/jira/browse/HDDS-1659 > Project: Hadoop Distributed Data Store > Issue Type: Task >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Labels: pull-request-available > Fix For: 0.4.1 > > Time Spent: 1h 50m > Remaining Estimate: 0h > > We think that it would be more effective to collect all the design docs in > one place and make it easier to review them by the community. > We propose to follow an approach where the proposals are committed to the > hadoop-hdds/docs project and the review can be the same as a review of a PR -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-1673) mapreduce smoketests are failig because an acl error
Elek, Marton created HDDS-1673:
--
Summary: mapreduce smoketests are failig because an acl error
Key: HDDS-1673
URL: https://issues.apache.org/jira/browse/HDDS-1673
Project: Hadoop Distributed Data Store
Issue Type: Bug
Reporter: Elek, Marton
After executing this command:
{code}
yarn jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.0.jar pi 3 3 2
{code}
The result is:
{code}
Number of Maps = 3
Samples per Map = 3
2019-06-12 03:16:20 ERROR OzoneClientFactory:294 - Couldn't create protocol
class org.apache.hadoop.ozone.client.rpc.RpcClient exception:
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at
org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:291)
at
org.apache.hadoop.ozone.client.OzoneClientFactory.getRpcClient(OzoneClientFactory.java:169)
at
org.apache.hadoop.fs.ozone.BasicOzoneClientAdapterImpl.(BasicOzoneClientAdapterImpl.java:134)
at
org.apache.hadoop.fs.ozone.OzoneClientAdapterImpl.(OzoneClientAdapterImpl.java:50)
at
org.apache.hadoop.fs.ozone.OzoneFileSystem.createAdapter(OzoneFileSystem.java:103)
at
org.apache.hadoop.fs.ozone.BasicOzoneFileSystem.initialize(BasicOzoneFileSystem.java:143)
at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3303)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124)
at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3352)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3320)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:479)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:227)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:463)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)
at
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.setInputPaths(FileInputFormat.java:522)
at
org.apache.hadoop.examples.QuasiMonteCarlo.estimatePi(QuasiMonteCarlo.java:275)
at
org.apache.hadoop.examples.QuasiMonteCarlo.run(QuasiMonteCarlo.java:360)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at
org.apache.hadoop.examples.QuasiMonteCarlo.main(QuasiMonteCarlo.java:368)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
at org.apache.hadoop.util.RunJar.main(RunJar.java:236)
Caused by: org.apache.hadoop.hdds.conf.ConfigurationException: Can't inject
configuration to class
org.apache.hadoop.ozone.security.acl.OzoneAclConfig.setUserDefaultRights
at
org.apache.hadoop.hdds.conf.OzoneConfiguration.getObject(OzoneConfiguration.java:160)
at
org.apache.hadoop.ozone.client.rpc.RpcClient.(RpcClient.java:148)
... 36 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.hdds.conf.OzoneConfiguration.getObject(OzoneConfiguration.java:137)
... 37 more
Caused by: java.lang.NullPointerException: Name is null
at java.lang.Enum.valueOf(Enum.java:236)
at
org.apache.hadoop.ozone.security.acl.IAccessAuthorizer$ACLType.valueOf(IAccessAuthorizer.java:48)
at
org.apache.hadoop.ozone.security.acl.OzoneAclConfig.setUserDefaultRights(OzoneAclConfig.java:43)
... 42 more
java.io.IOException: Could
[jira] [Created] (HDDS-1672) OzoneManager Lock change the volumeLock weight to 0
Bharat Viswanadham created HDDS-1672: Summary: OzoneManager Lock change the volumeLock weight to 0 Key: HDDS-1672 URL: https://issues.apache.org/jira/browse/HDDS-1672 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Bharat Viswanadham Assignee: Bharat Viswanadham Currently after acquiring volume lock, we cannot acquire user lock. This is causing an issue in Volume request implementation, acquire/release/reacquire volume lock. Case of Delete Volume Request: # Acquire volume lock. # Get Volume Info from DB # Release Volume lock. (We are releasing the lock, because while acquiring volume lock, we cannot acquire user lock0 # Get owner from volume Info read from DB # Acquire owner lock # Acquire volume lock # Do delete logic # release volume lock # release user lock We can avoid this acquire/release/reacquire lock issue by making volume lock as low weight. In this way, the above deleteVolume request will change as below # Acquire volume lock # Get Volume Info from DB # Get owner from volume Info read from DB # Acquire owner lock # Do delete logic # release owner lock # release volume lock. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-1671) Multiple unit test fails because of assertion while validating Acls
Mukul Kumar Singh created HDDS-1671: --- Summary: Multiple unit test fails because of assertion while validating Acls Key: HDDS-1671 URL: https://issues.apache.org/jira/browse/HDDS-1671 Project: Hadoop Distributed Data Store Issue Type: Bug Components: Security Reporter: Mukul Kumar Singh There are multiple unit test failures because of assertion in validateAcls https://builds.apache.org/job/hadoop-multibranch/job/PR-846/7/testReport/ -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-1670) Add limit support to /api/containers and /api/containers/{id} endpoints
Vivek Ratnavel Subramanian created HDDS-1670:
Summary: Add limit support to /api/containers and
/api/containers/{id} endpoints
Key: HDDS-1670
URL: https://issues.apache.org/jira/browse/HDDS-1670
Project: Hadoop Distributed Data Store
Issue Type: Task
Components: Ozone Recon
Affects Versions: 0.4.0
Reporter: Vivek Ratnavel Subramanian
Assignee: Vivek Ratnavel Subramanian
Add support for limit query param to limit the results of /api/containers and
/api/containers/\{id} endpoints
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-14561) dfs.datanode.data.dir does not honor file:/// scheme
Artem Ervits created HDFS-14561:
---
Summary: dfs.datanode.data.dir does not honor file:/// scheme
Key: HDFS-14561
URL: https://issues.apache.org/jira/browse/HDFS-14561
Project: Hadoop HDFS
Issue Type: Bug
Components: documentation, hdfs
Affects Versions: 2.8.5, 2.9.2
Reporter: Artem Ervits
_dfs.datanode.data.dir_ with {{file:///}} scheme is not being honored in Hadoop
2.8.5 and 2.9.2. I didn't test 3.x branches but good to validate also. In my
environment, only file:// seemed to work.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: [DISCUSS] A unified and open Hadoop community sync up schedule?
For Ozone, we have started using the Wiki itself as the agenda and after the meeting is over, we convert it into the meeting notes. Here is an example, the project owner can edit and maintain it, it is like 10 mins work - and allows anyone to add stuff into the agenda too. https://cwiki.apache.org/confluence/display/HADOOP/2019-06-10+Meeting+notes --Anu On Tue, Jun 11, 2019 at 10:20 AM Yufei Gu wrote: > +1 for this idea. Thanks Wangda for bringing this up. > > Some comments to share: > >- Agenda needed to be posted ahead of meeting and welcome any interested >party to contribute to topics. >- We should encourage more people to attend. That's whole point of the >meeting. >- Hopefully, this can mitigate the situation that some patches are >waiting for review for ever, which turns away new contributors. >- 30m per session sounds a little bit short, we can try it out and see >if extension is needed. > > Best, > > Yufei > > `This is not a contribution` > > > On Fri, Jun 7, 2019 at 4:39 PM Wangda Tan wrote: > > > Hi Hadoop-devs, > > > > Previous we have regular YARN community sync up (1 hr, biweekly, but not > > open to public). Recently because of changes in our schedules, Less folks > > showed up in the sync up for the last several months. > > > > I saw the K8s community did a pretty good job to run their sig meetings, > > there's regular meetings for different topics, notes, agenda, etc. Such > as > > > > > https://docs.google.com/document/d/13mwye7nvrmV11q9_Eg77z-1w3X7Q1GTbslpml4J7F3A/edit > > > > > > For Hadoop community, there are less such regular meetings open to the > > public except for Ozone project and offline meetups or Bird-of-Features > in > > Hadoop/DataWorks Summit. Recently we have a few folks joined DataWorks > > Summit at Washington DC and Barcelona, and lots (50+) of folks join the > > Ozone/Hadoop/YARN BoF, ask (good) questions and roadmaps. I think it is > > important to open such conversations to the public and let more > > folk/companies join. > > > > Discussed a small group of community members and wrote a short proposal > > about the form, time and topic of the community sync up, thanks for > > everybody who have contributed to the proposal! Please feel free to add > > your thoughts to the Proposal Google doc > > < > > > https://docs.google.com/document/d/1GfNpYKhNUERAEH7m3yx6OfleoF3MqoQk3nJ7xqHD9nY/edit# > > > > > . > > > > Especially for the following parts: > > - If you have interests to run any of the community sync-ups, please put > > your name to the table inside the proposal. We need more volunteers to > help > > run the sync-ups in different timezones. > > - Please add suggestions to the time, frequency and themes and feel free > to > > share your thoughts if we should do sync ups for other topics which are > not > > covered by the proposal. > > > > Link to the Proposal Google doc > > < > > > https://docs.google.com/document/d/1GfNpYKhNUERAEH7m3yx6OfleoF3MqoQk3nJ7xqHD9nY/edit# > > > > > > > Thanks, > > Wangda Tan > > >
Re: [DISCUSS] A unified and open Hadoop community sync up schedule?
+1 for this idea. Thanks Wangda for bringing this up. Some comments to share: - Agenda needed to be posted ahead of meeting and welcome any interested party to contribute to topics. - We should encourage more people to attend. That's whole point of the meeting. - Hopefully, this can mitigate the situation that some patches are waiting for review for ever, which turns away new contributors. - 30m per session sounds a little bit short, we can try it out and see if extension is needed. Best, Yufei `This is not a contribution` On Fri, Jun 7, 2019 at 4:39 PM Wangda Tan wrote: > Hi Hadoop-devs, > > Previous we have regular YARN community sync up (1 hr, biweekly, but not > open to public). Recently because of changes in our schedules, Less folks > showed up in the sync up for the last several months. > > I saw the K8s community did a pretty good job to run their sig meetings, > there's regular meetings for different topics, notes, agenda, etc. Such as > > https://docs.google.com/document/d/13mwye7nvrmV11q9_Eg77z-1w3X7Q1GTbslpml4J7F3A/edit > > > For Hadoop community, there are less such regular meetings open to the > public except for Ozone project and offline meetups or Bird-of-Features in > Hadoop/DataWorks Summit. Recently we have a few folks joined DataWorks > Summit at Washington DC and Barcelona, and lots (50+) of folks join the > Ozone/Hadoop/YARN BoF, ask (good) questions and roadmaps. I think it is > important to open such conversations to the public and let more > folk/companies join. > > Discussed a small group of community members and wrote a short proposal > about the form, time and topic of the community sync up, thanks for > everybody who have contributed to the proposal! Please feel free to add > your thoughts to the Proposal Google doc > < > https://docs.google.com/document/d/1GfNpYKhNUERAEH7m3yx6OfleoF3MqoQk3nJ7xqHD9nY/edit# > > > . > > Especially for the following parts: > - If you have interests to run any of the community sync-ups, please put > your name to the table inside the proposal. We need more volunteers to help > run the sync-ups in different timezones. > - Please add suggestions to the time, frequency and themes and feel free to > share your thoughts if we should do sync ups for other topics which are not > covered by the proposal. > > Link to the Proposal Google doc > < > https://docs.google.com/document/d/1GfNpYKhNUERAEH7m3yx6OfleoF3MqoQk3nJ7xqHD9nY/edit# > > > > Thanks, > Wangda Tan >
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1164/ [Jun 10, 2019 5:23:18 PM] (weichiu) HDFS-14494. Move Server logging of StatedId inside [Jun 10, 2019 7:18:15 PM] (weichiu) YARN-9471. Cleanup in TestLogAggregationIndexFileController. Contributed [Jun 10, 2019 8:45:37 PM] (weichiu) HDFS-10659. Namenode crashes after Journalnode re-installation in an HA [Jun 10, 2019 9:13:53 PM] (weichiu) HDFS-10210. Remove the defunct startKdc profile from hdfs. Contributed [Jun 10, 2019 9:33:24 PM] (sumasai) YARN-9569. Auto-created leaf queues do not honor cluster-wide min/max [Jun 11, 2019 12:21:52 AM] (weichiu) HDFS-14553. Make queue size of BlockReportProcessingThread configurable. [Jun 11, 2019 12:55:16 AM] (aengineer) HDFS-14234. Limit WebHDFS to specifc user, host, directory triples. -1 overall The following subsystems voted -1: asflicense findbugs hadolint pathlen unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: FindBugs : module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-documentstore Unread field:TimelineEventSubDoc.java:[line 56] Unread field:TimelineMetricSubDoc.java:[line 44] FindBugs : module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-mawo/hadoop-yarn-applications-mawo-core Class org.apache.hadoop.applications.mawo.server.common.TaskStatus implements Cloneable but does not define or use clone method At TaskStatus.java:does not define or use clone method At TaskStatus.java:[lines 39-346] Equals method for org.apache.hadoop.applications.mawo.server.worker.WorkerId assumes the argument is of type WorkerId At WorkerId.java:the argument is of type WorkerId At WorkerId.java:[line 114] org.apache.hadoop.applications.mawo.server.worker.WorkerId.equals(Object) does not check for null argument At WorkerId.java:null argument At WorkerId.java:[lines 114-115] Failed junit tests : hadoop.util.TestDiskChecker hadoop.hdfs.web.TestWebHdfsTimeouts hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade hadoop.yarn.applications.distributedshell.TestDistributedShell hadoop.mapreduce.v2.app.TestRuntimeEstimators hadoop.mapreduce.jobhistory.TestJobHistoryEventHandler hadoop.mapred.TestMRTimelineEventHandling hadoop.yarn.sls.appmaster.TestAMSimulator hadoop.ozone.container.common.impl.TestHddsDispatcher hadoop.ozone.container.ozoneimpl.TestOzoneContainer hadoop.ozone.container.common.statemachine.commandhandler.TestBlockDeletion hadoop.ozone.client.rpc.TestOzoneRpcClient hadoop.ozone.client.rpc.TestOzoneRpcClientWithRatis hadoop.ozone.client.rpc.TestSecureOzoneRpcClient hadoop.ozone.client.rpc.TestOzoneAtRestEncryption hadoop.fs.ozone.contract.ITestOzoneContractRootDir hadoop.fs.ozone.contract.ITestOzoneContractDistCp hadoop.fs.ozone.contract.ITestOzoneContractRename hadoop.fs.ozone.contract.ITestOzoneContractGetFileStatus hadoop.fs.ozone.contract.ITestOzoneContractOpen hadoop.fs.ozone.contract.ITestOzoneContractCreate hadoop.fs.ozone.contract.ITestOzoneContractSeek hadoop.fs.ozone.contract.ITestOzoneContractMkdir hadoop.fs.ozone.contract.ITestOzoneContractDelete hadoop.ozone.freon.TestFreonWithPipelineDestroy hadoop.ozone.freon.TestDataValidateWithUnsafeByteOperations hadoop.ozone.freon.TestFreonWithDatanodeRestart hadoop.ozone.freon.TestFreonWithDatanodeFastRestart hadoop.ozone.fsck.TestContainerMapper hadoop.ozone.freon.TestRandomKeyGenerator hadoop.ozone.freon.TestDataValidateWithSafeByteOperations cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1164/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1164/artifact/out/diff-compile-javac-root.txt [332K] checkstyle: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1164/artifact/out/diff-checkstyle-root.txt [17M] hadolint: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1164/artifact/out/diff-patch-hadolint.txt [8.0K] pathlen: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1164/artifact/out/pathlen.txt [12K] pylint: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1164/artifact/out/diff-patch-pylint.txt [120K] shellcheck: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux
[jira] [Created] (HDDS-1669) SCM startup is failing if network-topology-default.xml is part of a jar
Elek, Marton created HDDS-1669:
--
Summary: SCM startup is failing if network-topology-default.xml is
part of a jar
Key: HDDS-1669
URL: https://issues.apache.org/jira/browse/HDDS-1669
Project: Hadoop Distributed Data Store
Issue Type: Bug
Reporter: Elek, Marton
Assignee: Elek, Marton
network-topology-default.xml can be loaded from file or classpath. But the
NodeSchemaLoader assumes that the files on the classpath can be opened as a
file. It's true if the file is in etc/hadoop (which is part of the classpath)
but not true if the file is packaged to a jajr file:
{code}
scm_1 | 2019-06-11 13:18:03 INFO NodeSchemaLoader:118 - Loading file
from
jar:file:/opt/hadoop/share/ozone/lib/hadoop-hdds-common-0.5.0-SNAPSHOT.jar!/network-topology-default.xml
scm_1 | 2019-06-11 13:18:03 ERROR NodeSchemaManager:74 - Failed to load
schema file:network-topology-default.xml, error:
scm_1 | java.lang.IllegalArgumentException: URI is not hierarchical
scm_1 | at java.io.File.(File.java:418)
scm_1 | at
org.apache.hadoop.hdds.scm.net.NodeSchemaLoader.loadSchemaFromFile(NodeSchemaLoader.java:119)
scm_1 | at
org.apache.hadoop.hdds.scm.net.NodeSchemaManager.init(NodeSchemaManager.java:67)
scm_1 | at
org.apache.hadoop.hdds.scm.net.NetworkTopologyImpl.(NetworkTopologyImpl.java:63)
scm_1 | at
org.apache.hadoop.hdds.scm.server.StorageContainerManager.initializeSystemManagers(StorageContainerManager.java:382)
scm_1 | at
org.apache.hadoop.hdds.scm.server.StorageContainerManager.(StorageContainerManager.java:275)
scm_1 | at
org.apache.hadoop.hdds.scm.server.StorageContainerManager.(StorageContainerManager.java:208)
scm_1 | at
org.apache.hadoop.hdds.scm.server.StorageContainerManager.createSCM(StorageContainerManager.java:586)
scm_1 | at
org.apache.hadoop.hdds.scm.server.StorageContainerManagerStarter$SCMStarterHelper.start(StorageContainerManagerStarter.java:139)
scm_1 | at
org.apache.hadoop.hdds.scm.server.StorageContainerManagerStarter.startScm(StorageContainerManagerStarter.java:115)
scm_1 | at
org.apache.hadoop.hdds.scm.server.StorageContainerManagerStarter.call(StorageContainerManagerStarter.java:67)
scm_1 | at
org.apache.hadoop.hdds.scm.server.StorageContainerManagerStarter.call(StorageContainerManagerStarter.java:42)
scm_1 | at picocli.CommandLine.execute(CommandLine.java:1173)
scm_1 | at picocli.CommandLine.access$800(CommandLine.java:141)
scm_1 | at
picocli.CommandLine$RunLast.handle(CommandLine.java:1367)
scm_1 | at
picocli.CommandLine$RunLast.handle(CommandLine.java:1335)
scm_1 | at
picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:1243)
scm_1 | at
picocli.CommandLine.parseWithHandlers(CommandLine.java:1526)
scm_1 | at
picocli.CommandLine.parseWithHandler(CommandLine.java:1465)
scm_1 | at
org.apache.hadoop.hdds.cli.GenericCli.execute(GenericCli.java:65)
scm_1 | at
org.apache.hadoop.hdds.cli.GenericCli.run(GenericCli.java:56)
scm_1 | at
org.apache.hadoop.hdds.scm.server.StorageContainerManagerStarter.main(StorageContainerManagerStarter.java:56)
scm_1 | Failed to load schema file:network-topology-default.xml, error:
{code}
The quick fix is to keep the current behaviour but read the file from
classloader.getResourceAsStream() instead of classloader.getResource().toURI()
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/349/ No changes -1 overall The following subsystems voted -1: asflicense findbugs hadolint pathlen unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: XML : Parsing Error(s): hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/empty-configuration.xml hadoop-tools/hadoop-azure/src/config/checkstyle-suppressions.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/public/crossdomain.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/public/crossdomain.xml FindBugs : module:hadoop-common-project/hadoop-common Class org.apache.hadoop.fs.GlobalStorageStatistics defines non-transient non-serializable instance field map In GlobalStorageStatistics.java:instance field map In GlobalStorageStatistics.java FindBugs : module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client Boxed value is unboxed and then immediately reboxed in org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnRWHelper.readResultsWithTimestamps(Result, byte[], byte[], KeyConverter, ValueConverter, boolean) At ColumnRWHelper.java:then immediately reboxed in org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnRWHelper.readResultsWithTimestamps(Result, byte[], byte[], KeyConverter, ValueConverter, boolean) At ColumnRWHelper.java:[line 335] Failed junit tests : hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys hadoop.hdfs.web.TestWebHdfsTimeouts hadoop.registry.secure.TestSecureLogins hadoop.yarn.server.timelineservice.security.TestTimelineAuthFilterForV2 cc: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/349/artifact/out/diff-compile-cc-root-jdk1.7.0_95.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/349/artifact/out/diff-compile-javac-root-jdk1.7.0_95.txt [328K] cc: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/349/artifact/out/diff-compile-cc-root-jdk1.8.0_212.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/349/artifact/out/diff-compile-javac-root-jdk1.8.0_212.txt [308K] checkstyle: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/349/artifact/out/diff-checkstyle-root.txt [16M] hadolint: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/349/artifact/out/diff-patch-hadolint.txt [4.0K] pathlen: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/349/artifact/out/pathlen.txt [12K] pylint: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/349/artifact/out/diff-patch-pylint.txt [24K] shellcheck: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/349/artifact/out/diff-patch-shellcheck.txt [72K] shelldocs: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/349/artifact/out/diff-patch-shelldocs.txt [8.0K] whitespace: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/349/artifact/out/whitespace-eol.txt [12M] https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/349/artifact/out/whitespace-tabs.txt [1.2M] xml: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/349/artifact/out/xml.txt [12K] findbugs: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/349/artifact/out/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html [8.0K] https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/349/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase_hadoop-yarn-server-timelineservice-hbase-client-warnings.html [8.0K] javadoc: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/349/artifact/out/diff-javadoc-javadoc-root-jdk1.7.0_95.txt [16K] https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/349/artifact/out/diff-javadoc-javadoc-root-jdk1.8.0_212.txt [1.1M] unit: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/349/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [284K] https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/349/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-registry.txt [12K] https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/349/art
[jira] [Created] (HDFS-14560) Allow block replication parameters to be refreshable
Stephen O'Donnell created HDFS-14560:
Summary: Allow block replication parameters to be refreshable
Key: HDFS-14560
URL: https://issues.apache.org/jira/browse/HDFS-14560
Project: Hadoop HDFS
Issue Type: Improvement
Components: namenode
Affects Versions: 3.3.0
Reporter: Stephen O'Donnell
Assignee: Stephen O'Donnell
There are 3 key parameters that control the speed of block replication across
the cluster:
{code}
dfs.namenode.replication.max-streams
dfs.namenode.replication.max-streams-hard-limit
dfs.namenode.replication.work.multiplier.per.iteration
{code}
These are used when decomissioning nodes and when under replicated blocks are
being recovered across the cluster. There are times when it may be desirable to
increase the speed of replication and then reduce it again (eg during off peak
hours) without restarting the namenode.
Therefore it would be good to be able to reconfigure / refresh these parameters
dynamically without a namenode restart.
This Jira is to allow these parameters to be refreshed at runtime without a
restart.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-1668) Add liveness probe to the example k8s resources files
Elek, Marton created HDDS-1668: -- Summary: Add liveness probe to the example k8s resources files Key: HDDS-1668 URL: https://issues.apache.org/jira/browse/HDDS-1668 Project: Hadoop Distributed Data Store Issue Type: Sub-task Reporter: Elek, Marton Assignee: Elek, Marton In kubernetes resources we can define livebess probes which can help to detect any failure. If the define port is not available the pod will be rescheduled. We need to add the liveness probes to our k8s resource files. Note: We shouldn't add readiness probes. Readiness probe is about the service availability. The service/dns can be available only after the service is restarted. This is not good for us as: * We need DNS resolution during the startup (See OzoneManager.loadOMHAConfigs) * We already implemented retry in case of missing DNS entries -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-14559) Optimizing safemode leave mechanism
He Xiaoqiao created HDFS-14559: -- Summary: Optimizing safemode leave mechanism Key: HDFS-14559 URL: https://issues.apache.org/jira/browse/HDFS-14559 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Reporter: He Xiaoqiao Assignee: He Xiaoqiao As HDFS-14186 mentioned, The last stage of namenode startup, it will leave safemode based on the condition that if blocks num reach to threshold. However the current condition is complete based on total blocks rather than total replications. So for a large cluster, after total blocks has reported from datanode, there are still large block replication pending report and load of namenode is continue high for long times. In some extreme case, between leave safemode time and process block report completely, namenode will not provide normal service and some datanodes could dead then register/blockreport again and again. In one word, we need to upgrade safemode leave mechanism to support large cluster restart smooth. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-14424) NN failover failed beacuse of lossing paxos directory
[ https://issues.apache.org/jira/browse/HDFS-14424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] star resolved HDFS-14424. - Resolution: Duplicate > NN failover failed beacuse of lossing paxos directory > - > > Key: HDFS-14424 > URL: https://issues.apache.org/jira/browse/HDFS-14424 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: star >Assignee: star >Priority: Major > > Recently, our hadoop namenode shutdown when switching active namenode, just > because of missing paxos directory. It is created in the default /tmp path > and deleted by os for no operation in 7 days. We can avoid this by moving > journal directory to a none tmp dir, but it‘s better to make sure namenode > works well by a default config. > The issue throws exception similar to HDFS-10659, also caused by missing > paxos directory. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: [VOTE] Merge HDFS-13891(RBF) to trunk
+1(non-binding). Regards, - Takanobu From: Xiaoqiao He Sent: Monday, June 10, 2019 15:47 To: Ranith Sardar Cc: Brahma Reddy Battula; Hadoop Common; Hdfs-dev Subject: Re: [VOTE] Merge HDFS-13891(RBF) to trunk +1 (non-binding) - Try to merge branch HDFS-13891(RBF) to trunk at local and no conflict or failure. - Built from merged sources. - Ran 85 RBF test class at local and result shows: (Tests run: 639, Failures: 2, Errors: 2, Skipped: 2) - failed tests include #TestRouterWithSecureStartup and #TestRouterHttpDelegationToken. I don't think it is blocked issue. Thanks Brahma for organizing the merge. On Mon, Jun 10, 2019 at 12:27 PM Ranith Sardar wrote: > +1 (Non-binding) > > -Original Message- > From: Brahma Reddy Battula [mailto:bra...@apache.org] > Sent: 09 June 2019 19:31 > To: Hadoop Common ; Hdfs-dev < > hdfs-dev@hadoop.apache.org> > Subject: [VOTE] Merge HDFS-13891(RBF) to trunk > > Updated mail... > > -- Forwarded message - > From: Brahma Reddy Battula > Date: Sun, Jun 9, 2019 at 7:26 PM > Subject: Re: [VOTE] Merge HDFS-13891(RBF) to trunk > To: Xiaoqiao He > Cc: Akira Ajisaka , Chittaranjan Hota < > chitts.h...@gmail.com>, Giovanni Matteo Fumarola < > giovanni.fumar...@gmail.com>, Hadoop Common , > Hdfs-dev , Iñigo Goiri > > > Hi All, > > Given the positive response to the discussion thread [1], here is the > formal vote thread to merge HDFS-13891 in to trunk. > > Summary of code changes: > 1. Code changes for this branch are done in the hadoop-hdfs-rbf > subproject, there is no impact to hadoop-hdfs and hadoop-common. > 2. Added Security support for RBF > 3. Added Missing Client Protocol API's > 4. Bug Fixes/ Improvments > > > The vote will run for 7 days, ending Sat June 15th. I will start this > vote with my +1. > > Regards, > Brahma Reddy Battula > > 1). > > https://lists.apache.org/thread.html/cdc2e084874b30bf6af2dd827bcbdba4ab8d3d983a8b9796e61608e6@%3Ccommon-dev.hadoop.apache.org%3E > > > On Wed, Jun 5, 2019 at 3:44 PM Xiaoqiao He wrote: > > > Thanks Brahma for starting the thread. > > +1 for merging. > > > > He Xiaoqiao > > > > On Tue, Jun 4, 2019 at 1:53 PM Chittaranjan Hota > > > > wrote: > > > >> Thanks Brahma initiating this. > >> +1(non-binding) for merge. > >> > >> @Uber we have almost all changes specially rbf security in production > >> for a while now without issues. > >> > >> On Mon, Jun 3, 2019 at 12:56 PM Giovanni Matteo Fumarola < > >> giovanni.fumar...@gmail.com> wrote: > >> > >> > +1 on merging. > >> > > >> > Thanks Brahma for starting the thread. > >> > > >> > On Mon, Jun 3, 2019 at 10:00 AM Iñigo Goiri > wrote: > >> > > >> > > Thank you Brahma for pushing this. > >> > > > >> > > As you mentioned, we have already taken most of the changes into > >> > > production. > >> > > I want to highlight that the main contribution is the addition of > >> > security. > >> > > We have been able to test this at a smaller scale (~500 servers > >> > > and 4 > >> > > subclusters) and the performance is great with our current > >> > > ZooKeeper deployment. > >> > > I would also like to highlight that all the changes are > >> > > constrained to hadoop-hdfs-rbf and there is no differences in > commons or HDFS. > >> > > > >> > > +1 on merging > >> > > > >> > > Inigo > >> > > > >> > > On Sun, Jun 2, 2019 at 10:19 PM Akira Ajisaka > >> > > > >> > wrote: > >> > > > >> > > > Thanks Brahma for starting the discussion. > >> > > > I'm +1 for merging this. > >> > > > > >> > > > FYI: In Yahoo! JAPAN, deployed all these changes in 20 nodes > >> > > > cluster with 2 routers (not in production) and running several > tests. > >> > > > > >> > > > Regards, > >> > > > Akira > >> > > > > >> > > > On Sun, Jun 2, 2019 at 12:40 PM Brahma Reddy Battula < > >> > bra...@apache.org> > >> > > > wrote: > >> > > > > > >> > > > > Dear Hadoop Developers > >> > > > > > >> > > > > I would like to propose RBF Branch (HDFS-13891) merge into > trunk. > >> We > >> > > have > >> > > > > been working on this feature from last several months. > >> > > > > This feature work received the contributions from different > >> > companies. > >> > > > All > >> > > > > of the feature development happened smoothly and > >> > > > > collaboratively > >> in > >> > > > JIRAs. > >> > > > > > >> > > > > Kindly do take a look at the branch and raise issues/concerns > >> > > > > that > >> > need > >> > > > to > >> > > > > be addressed before the merge. > >> > > > > > >> > > > > *Highlights of HDFS-13891 Branch:* > >> > > > > = > >> > > > > > >> > > > > Adding Security to RBF(1) > >> > > > > Adding Missing Client API's(2) Improvements/Bug Fixing > >> > > > > Critical - HDFS-13637, HDFS-13834 > >> > > > > > >> > > > > *Commits:* > >> > > > > > >> > > > > > >> > > > > No of JIRAs Resolved: 72 > >> > > > > > >> > > > > All this commits are in RBF Module. No changes in hdfs/common. > >> > > > > > >> > > > > *Test
