Re: [VOTE] Merging branch HDFS-8707 (native HDFS client) to trunk
+1 (binding) Thanks, Mukul On 14/03/18, 2:06 AM, "Owen O'Malley" wrote: +1 (binding) .. Owen On Sun, Mar 11, 2018 at 6:20 PM, Chris Douglas wrote: > +1 (binding) -C > > On Thu, Mar 8, 2018 at 9:31 AM, Jim Clampffer > wrote: > > Hi Everyone, > > > > The feedback was generally positive on the discussion thread [1] so I'd > > like to start a formal vote for merging HDFS-8707 (libhdfs++) into trunk. > > The vote will be open for 7 days and end 6PM EST on 3/15/18. > > > > This branch includes a C++ implementation of an HDFS client for use in > > applications that don't run an in-process JVM. Right now the branch only > > supports reads and metadata calls. > > > > Features (paraphrasing the list from the discussion thread): > > -Avoiding the JVM means applications that use libhdfs++ can explicitly > > control resources (memory, FDs, threads). The driving goal for this > > project was to let C/C++ applications access HDFS while maintaining a > > single heap. > > -Includes support for Kerberos authentication. > > -Includes a libhdfs/libhdfs3 compatible C API as well as a C++ API that > > supports asynchronous operations. Applications that only do reads may be > > able to use this as a drop in replacement for libhdfs. > > -Asynchronous IO is built on top of boost::asio which in turn uses > > select/epoll so many sockets can be monitored from a single thread (or > > thread pool) rather than spawning a thread to sleep on a blocked socket. > > -Includes a set of utilities written in C++ that mirror the CLI tools > (e.g. > > ./hdfs dfs -ls). These have a 3 order of magnitude lower startup time > than > > java client which is useful for scripts that need to work with many > files. > > -Support for cancelable reads that release associated resources > > immediately. Useful for applications that need to be responsive to > > interactive users. > > > > Other points: > > -This is almost all new code in a new subdirectory. No Java source for > the > > rest of hadoop was changed so there's no risk of regressions there. The > > only changes outside of that subdirectory were integrating the build in > > some of the pom files and adding a couple dependencies to the DockerFile. > > -The library has had plenty of burn-in time. It's been used in > production > > for well over a year and is indirectly being distributed as part of the > > Apache ORC project (in the form of a third party dependency). > > -There isn't much in the way of well formatted documentation right now. > > The documentation for the libhdfs API is applicable to the libhdfs++ C > API. > > Header files describe various component including details about threading > > and lifecycle expectations for important objects. Good places to start > are > > hdfspp.h, filesystem.h, filehandle.h, rpc_connection.h and rpc_enginel.h. > > > > I'll start with my +1 (binding). > > > > [1] > > http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-dev/ > 201803.mbox/browser > > (second message in thread, can't figure out how to link directly to mine) > > > > Thanks! > > - > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org > >
[jira] [Created] (HDFS-13278) Correct the logic of mount validate to avoid the bad mountPoint
maobaolong created HDFS-13278: - Summary: Correct the logic of mount validate to avoid the bad mountPoint Key: HDFS-13278 URL: https://issues.apache.org/jira/browse/HDFS-13278 Project: Hadoop HDFS Issue Type: Sub-task Components: hdfs Affects Versions: 3.2.0 Reporter: maobaolong -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13277) Improve move to account for usage (number of files) to limit trash dir size
Bharat Viswanadham created HDFS-13277: - Summary: Improve move to account for usage (number of files) to limit trash dir size Key: HDFS-13277 URL: https://issues.apache.org/jira/browse/HDFS-13277 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Bharat Viswanadham Assignee: Bharat Viswanadham The trash subdirectory maximum entries. This puts an upper limit on the size of subdirectories in replica-trash. Set this default value to blockinvalidateLimit. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13276) TestDistributedFileSystem doesn't cleanup MiniDFSCluster if test times out
Íñigo Goiri created HDFS-13276: -- Summary: TestDistributedFileSystem doesn't cleanup MiniDFSCluster if test times out Key: HDFS-13276 URL: https://issues.apache.org/jira/browse/HDFS-13276 Project: Hadoop HDFS Issue Type: Improvement Reporter: Íñigo Goiri Assignee: Lukas Majercak If a unit tests times out, it may leave a MiniDFSCluster behing. This is particularly bad in Windows where the new MiniDFSCluster cannot start and all tests will fail after this one. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13275) Adding logs for BlockPoolManager#refreshNamenodes
Xiaoyu Yao created HDFS-13275: - Summary: Adding logs for BlockPoolManager#refreshNamenodes Key: HDFS-13275 URL: https://issues.apache.org/jira/browse/HDFS-13275 Project: Hadoop HDFS Issue Type: Improvement Reporter: Xiaoyu Yao Assignee: Ajay Kumar This helps detect and troubleshooting misconfigured NN service RPC or NN lifeline RPC address. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: [VOTE] Merging branch HDFS-8707 (native HDFS client) to trunk
+1 (binding) .. Owen On Sun, Mar 11, 2018 at 6:20 PM, Chris Douglas wrote: > +1 (binding) -C > > On Thu, Mar 8, 2018 at 9:31 AM, Jim Clampffer > wrote: > > Hi Everyone, > > > > The feedback was generally positive on the discussion thread [1] so I'd > > like to start a formal vote for merging HDFS-8707 (libhdfs++) into trunk. > > The vote will be open for 7 days and end 6PM EST on 3/15/18. > > > > This branch includes a C++ implementation of an HDFS client for use in > > applications that don't run an in-process JVM. Right now the branch only > > supports reads and metadata calls. > > > > Features (paraphrasing the list from the discussion thread): > > -Avoiding the JVM means applications that use libhdfs++ can explicitly > > control resources (memory, FDs, threads). The driving goal for this > > project was to let C/C++ applications access HDFS while maintaining a > > single heap. > > -Includes support for Kerberos authentication. > > -Includes a libhdfs/libhdfs3 compatible C API as well as a C++ API that > > supports asynchronous operations. Applications that only do reads may be > > able to use this as a drop in replacement for libhdfs. > > -Asynchronous IO is built on top of boost::asio which in turn uses > > select/epoll so many sockets can be monitored from a single thread (or > > thread pool) rather than spawning a thread to sleep on a blocked socket. > > -Includes a set of utilities written in C++ that mirror the CLI tools > (e.g. > > ./hdfs dfs -ls). These have a 3 order of magnitude lower startup time > than > > java client which is useful for scripts that need to work with many > files. > > -Support for cancelable reads that release associated resources > > immediately. Useful for applications that need to be responsive to > > interactive users. > > > > Other points: > > -This is almost all new code in a new subdirectory. No Java source for > the > > rest of hadoop was changed so there's no risk of regressions there. The > > only changes outside of that subdirectory were integrating the build in > > some of the pom files and adding a couple dependencies to the DockerFile. > > -The library has had plenty of burn-in time. It's been used in > production > > for well over a year and is indirectly being distributed as part of the > > Apache ORC project (in the form of a third party dependency). > > -There isn't much in the way of well formatted documentation right now. > > The documentation for the libhdfs API is applicable to the libhdfs++ C > API. > > Header files describe various component including details about threading > > and lifecycle expectations for important objects. Good places to start > are > > hdfspp.h, filesystem.h, filehandle.h, rpc_connection.h and rpc_enginel.h. > > > > I'll start with my +1 (binding). > > > > [1] > > http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-dev/ > 201803.mbox/browser > > (second message in thread, can't figure out how to link directly to mine) > > > > Thanks! > > - > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org > >
[jira] [Created] (HDFS-13274) RBF: Extend RouterRpcClient to use multiple sockets
Íñigo Goiri created HDFS-13274: -- Summary: RBF: Extend RouterRpcClient to use multiple sockets Key: HDFS-13274 URL: https://issues.apache.org/jira/browse/HDFS-13274 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Íñigo Goiri -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13273) Fix compilation issue in trunk
Bharat Viswanadham created HDFS-13273: - Summary: Fix compilation issue in trunk Key: HDFS-13273 URL: https://issues.apache.org/jira/browse/HDFS-13273 Project: Hadoop HDFS Issue Type: Bug Reporter: Bharat Viswanadham Assignee: Bharat Viswanadham [ERROR] /home/jenkins/jenkins-slave/workspace/Hadoop-trunk-Commit/source/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileStatusWithECPolicy.java:[40,8] class TestFileStatusWithDefaultECPolicy is public, should be declared in a file named TestFileStatusWithDefaultECPolicy.java -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13272) DataNodeHttpServer hard-codes HttpServer2 threads at 10
Erik Krogen created HDFS-13272: -- Summary: DataNodeHttpServer hard-codes HttpServer2 threads at 10 Key: HDFS-13272 URL: https://issues.apache.org/jira/browse/HDFS-13272 Project: Hadoop HDFS Issue Type: Bug Components: datanode Reporter: Erik Krogen Assignee: Erik Krogen In HDFS-7279, the Jetty server on the DataNode was hard-coded to use 10 threads. In addition to the possibility of this being too few threads, it is much higher than necessary in resource constrained environments such as MiniDFSCluster. To avoid compatibility issues, rather than using {{HttpServer2#HTTP_MAX_THREADS}} directly, we can introduce a new configuration for the DataNode's thread pool size. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/720/ [Mar 12, 2018 2:01:42 PM] (stevel) HADOOP-15297. Make S3A etag => checksum feature optional. Contributed by [Mar 12, 2018 4:58:56 PM] (inigoiri) HDFS-11399. Many tests fails in Windows due to injecting disk failures. [Mar 12, 2018 7:07:22 PM] (aengineer) HDFS-10618. TestPendingReconstruction#testPendingAndInvalidate is flaky [Mar 12, 2018 8:43:27 PM] (cdouglas) HADOOP-14742. Document multi-URI replication Inode for ViewFS. [Mar 12, 2018 9:29:44 PM] (cdouglas) HDFS-12677. Extend TestReconstructStripedFile with a random EC policy. [Mar 12, 2018 11:35:26 PM] (yufei) YARN-8024. LOG in class MaxRunningAppsEnforcer is initialized with a [Mar 13, 2018 12:28:15 AM] (inigoiri) HDFS-13241. RBF: TestRouterSafemode failed if the port is in use. -1 overall The following subsystems voted -1: findbugs unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: FindBugs : module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api org.apache.hadoop.yarn.api.records.Resource.getResources() may expose internal representation by returning Resource.resources At Resource.java:by returning Resource.resources At Resource.java:[line 234] FindBugs : module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common Nullcheck of callerUGI at line 146 of value previously dereferenced in org.apache.hadoop.yarn.server.webapp.AppBlock.render(HtmlBlock$Block) At AppBlock.java:146 of value previously dereferenced in org.apache.hadoop.yarn.server.webapp.AppBlock.render(HtmlBlock$Block) At AppBlock.java:[line 146] Failed junit tests : hadoop.hdfs.web.TestWebHdfsTimeouts hadoop.hdfs.server.balancer.TestBalancerRPCDelay hadoop.hdfs.TestDFSStripedOutputStreamWithFailure hadoop.hdfs.server.federation.router.TestRouterSafemode hadoop.yarn.server.nodemanager.webapp.TestContainerLogsPage hadoop.yarn.applications.distributedshell.TestDistributedShell hadoop.yarn.sls.TestSLSStreamAMSynth cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/720/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/720/artifact/out/diff-compile-javac-root.txt [292K] checkstyle: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/720/artifact/out/diff-checkstyle-root.txt [17M] pylint: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/720/artifact/out/diff-patch-pylint.txt [24K] shellcheck: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/720/artifact/out/diff-patch-shellcheck.txt [20K] shelldocs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/720/artifact/out/diff-patch-shelldocs.txt [12K] whitespace: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/720/artifact/out/whitespace-eol.txt [9.2M] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/720/artifact/out/whitespace-tabs.txt [288K] xml: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/720/artifact/out/xml.txt [4.0K] findbugs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/720/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-warnings.html [8.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/720/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common-warnings.html [8.0K] javadoc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/720/artifact/out/diff-javadoc-javadoc-root.txt [760K] unit: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/720/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [328K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/720/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt [48K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/720/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-applications-distributedshell.txt [12K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/720/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt [84K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/720/artifact/out/patch-unit-hadoop-tools_hadoop-sls.txt [12K] Powered by Apache Yetus 0.8.0-SNAPSHOT http://yetus.apa
[jira] [Created] (HDFS-13271) WebHDFS: Add constructor in SnapshottableDirectoryStatus with HdfsFileStatus as argument
Lokesh Jain created HDFS-13271: -- Summary: WebHDFS: Add constructor in SnapshottableDirectoryStatus with HdfsFileStatus as argument Key: HDFS-13271 URL: https://issues.apache.org/jira/browse/HDFS-13271 Project: Hadoop HDFS Issue Type: Bug Components: webhdfs Reporter: Lokesh Jain Assignee: Lokesh Jain This jira aims to add a constructor in SnapshottableDirectoryStatus which takes HdfsFileStatus as a argument. This constructor will be used in JsonUtilClient#toSnapshottableDirectoryStatus for creating a SnapshottableDirectoryStatus object. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13270) RBF: Router audit logger
maobaolong created HDFS-13270: - Summary: RBF: Router audit logger Key: HDFS-13270 URL: https://issues.apache.org/jira/browse/HDFS-13270 Project: Hadoop HDFS Issue Type: New Feature Components: hdfs Affects Versions: 3.2.0 Reporter: maobaolong We can use router auditlogger to log the client info and cmd, because the FSNamesystem#Auditlogger's log think the client are all from router. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org