[jira] [Created] (HADOOP-14994) Aliyun:Improve oss error logging
Genmao Yu created HADOOP-14994: -- Summary: Aliyun:Improve oss error logging Key: HADOOP-14994 URL: https://issues.apache.org/jira/browse/HADOOP-14994 Project: Hadoop Common Issue Type: Sub-task Components: fs/oss Affects Versions: 3.0.0-beta1 Reporter: Genmao Yu -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14995) Configuration should load resource by forced as option
DENG FEI created HADOOP-14995: - Summary: Configuration should load resource by forced as option Key: HADOOP-14995 URL: https://issues.apache.org/jira/browse/HADOOP-14995 Project: Hadoop Common Issue Type: Improvement Components: conf Reporter: DENG FEI Priority: Critical MapReduce Job, the task initialization by _job.xml_ , but if missing the soft link of "job.xml" file,will do nothing under the quit mode of Configuration. map tasks as direct output to HDFS without reduce tasks, if map task missing job.xml, it's will run with default behavior, just as _IdentityMapper_ and output to local. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk
Hi Konstantin, Thank you for taking out time to review ozone. I appreciate your comments and questions. > There are two main limitations in HDFS > a) The throughput of Namespace operations. Which is limited by the >number of RPCs the NameNode can handle > b) The number of objects (files + blocks) the system can maintain. >Which is limited by the memory size of the NameNode. I agree completely. We believe ozone attempts to address both these issues for HDFS. Let us look at the Number of objects problem. Ozone directly addresses the scalability of number of blocks by introducing storage containers that can hold multiple blocks together. The earlier efforts on this were complicated by the fact that block manager and namespace are intertwined in HDFS Namenode. There have been efforts in past to separate block manager from namespace for e.g. HDFS-5477. Ozone addresses this problem by cleanly separating the block layer. Separation of block layer also addresses the file/directories scalability because it frees up the blockmap from the namenode. Separate block layer relieves namenode from handling block reports, IBRs, heartbeats, replication monitor etc, and thus reduces the contention on FSNamesystem lock and significantly reduces the GC pressure on the namenode. These improvements will greatly help the RPC performance of the Namenode. > Ozone is probably just the first step in rebuilding HDFS under a new > architecture. With the next steps presumably being HDFS-10419 and > HDFS-8. The design doc for the new architecture has never been > published. We do believe that Namenode can leverage the ozone’s storage container layer, however, that is also a big effort. We would like to first have block layer stabilized in ozone before taking that up. However, we would certainly support any community effort on that, and in fact it was brought up in last BoF session at the summit. Big data is evolving rapidly. We see our customers needing scalable file systems, Objects stores(like S3) and Block Store(for docker and VMs). Ozone improves HDFS in two ways. It addresses throughput and scale issues of HDFS, and enriches it with newer capabilities. > Ozone is a big enough system to deserve its own project. I took a quick look at the core code in ozone and the cloc command reports 22,511 lines of functionality changes in Java. This patch also brings in web framework code like Angular.js and that brings in bunch of css and js files that contribute to the size of the patch, and the rest are test and documentation changes. I hope this addresses your concerns. Best regards, jitendra On 10/28/17, 2:00 PM, "Konstantin Shvachko" wrote: Hey guys, It is an interesting question whether Ozone should be a part of Hadoop. There are two main reasons why I think it should not. 1. With close to 500 sub-tasks, with 6 MB of code changes, and with a sizable community behind, it looks to me like a whole new project. It is essentially a new storage system, with different (than HDFS) architecture, separate S3-like APIs. This is really great - the World sure needs more distributed file systems. But it is not clear why Ozone should co-exist with HDFS under the same roof. 2. Ozone is probably just the first step in rebuilding HDFS under a new architecture. With the next steps presumably being HDFS-10419 and HDFS-8. The design doc for the new architecture has never been published. I can only assume based on some presentations and personal communications that the idea is to use Ozone as a block storage, and re-implement NameNode, so that it stores only a partial namesapce in memory, while the bulk of it (cold data) is persisted to a local storage. Such architecture makes me wonder if it solves Hadoop's main problems. There are two main limitations in HDFS: a. The throughput of Namespace operations. Which is limited by the number of RPCs the NameNode can handle b. The number of objects (files + blocks) the system can maintain. Which is limited by the memory size of the NameNode. The RPC performance (a) is more important for Hadoop scalability than the object count (b). The read RPCs being the main priority. The new architecture targets the object count problem, but in the expense of the RPC throughput. Which seems to be a wrong resolution of the tradeoff. Also based on the use patterns on our large clusters we read up to 90% of the data we write, so cold data is a small fraction and most of it must be cached. To summarize: - Ozone is a big enough system to deserve its own project. - The architecture that Ozone leads to does not seem to solve the intrinsic problems of current HDFS. I will post my opinion in the Ozone jira. Should be more convenient to discuss it there for further reference. Thanks,
Cutting branch-2.9
We want to give heads up that we are going to cut branch-2.9 tomorrow morning. We are down to 3 blockers and they all are close to being committed (thanks everyone): https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.9+Release There are 4 other non-blocker JIRAs that are targeted for 2.9.0 which are close to completion. Folks who are working/reviewing these, kindly prioritize accordingly so that we can make the release on time. https://issues.apache.org/jira/browse/YARN-7398?filter=12342468 Thanks in advance! -Subru/Arun
[VOTE] Merge yarn-native-services branch into trunk
Hi All, I would like to restart the vote for merging yarn-native-services to trunk. Since last vote, we have been working on several issues in documentation, DNS, CLI modifications etc. We believe now the feature is in a much better shape. Some back ground: At a high level, the following are the key feautres implemented. - YARN-5079[1]. A native YARN framework (ApplicationMaster) to orchestrate existing services to YARN either docker or non-docker based. - YARN-4793[2]. A Rest API service embeded in RM (optional) for user to deploy a service via a simple JSON spec - YARN-4757[3]. Extending today's service registry with a simple DNS service to enable users to discover services deployed on YARN via standard DNS lookup - YARN-6419[4]. UI support for native-services on the new YARN UI All these new services are optional and are sitting outside of the existing system, and have no impact on existing system if disabled. Special thanks to a team of folks who worked hard towards this: Billie Rinaldi, Gour Saha, Vinod Kumar Vavilapalli, Jonathan Maron, Rohith Sharma K S, Sunil G, Akhil PB, Eric Yang. This effort could not be possible without their ideas and hard work. Also thanks Allen for some review and verifications. Thanks, Jian [1] https://issues.apache.org/jira/browse/YARN-5079 [2] https://issues.apache.org/jira/browse/YARN-4793 [3] https://issues.apache.org/jira/browse/YARN-4757 [4] https://issues.apache.org/jira/browse/YARN-6419
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/574/ [Oct 29, 2017 11:44:16 PM] (yufei) YARN-6747. TestFSAppStarvation.testPreemptionEnable fails [Oct 30, 2017 1:54:33 AM] (templedf) YARN-7374. Improve performance of DRF comparisons for resource types in -1 overall The following subsystems voted -1: asflicense findbugs unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: FindBugs : module:hadoop-tools/hadoop-resourceestimator Dead store to jobHistory in org.apache.hadoop.resourceestimator.service.ResourceEstimatorService.getHistoryResourceSkyline(String, String) At ResourceEstimatorService.java:org.apache.hadoop.resourceestimator.service.ResourceEstimatorService.getHistoryResourceSkyline(String, String) At ResourceEstimatorService.java:[line 196] Incorrect lazy initialization and update of static field org.apache.hadoop.resourceestimator.service.ResourceEstimatorService.skylineStore in new org.apache.hadoop.resourceestimator.service.ResourceEstimatorService() At ResourceEstimatorService.java:of static field org.apache.hadoop.resourceestimator.service.ResourceEstimatorService.skylineStore in new org.apache.hadoop.resourceestimator.service.ResourceEstimatorService() At ResourceEstimatorService.java:[lines 78-82] Write to static field org.apache.hadoop.resourceestimator.service.ResourceEstimatorService.config from instance method new org.apache.hadoop.resourceestimator.service.ResourceEstimatorService() At ResourceEstimatorService.java:from instance method new org.apache.hadoop.resourceestimator.service.ResourceEstimatorService() At ResourceEstimatorService.java:[line 80] Write to static field org.apache.hadoop.resourceestimator.service.ResourceEstimatorService.gson from instance method new org.apache.hadoop.resourceestimator.service.ResourceEstimatorService() At ResourceEstimatorService.java:from instance method new org.apache.hadoop.resourceestimator.service.ResourceEstimatorService() At ResourceEstimatorService.java:[line 106] Write to static field org.apache.hadoop.resourceestimator.service.ResourceEstimatorService.logParser from instance method new org.apache.hadoop.resourceestimator.service.ResourceEstimatorService() At ResourceEstimatorService.java:from instance method new org.apache.hadoop.resourceestimator.service.ResourceEstimatorService() At ResourceEstimatorService.java:[line 86] Write to static field org.apache.hadoop.resourceestimator.service.ResourceEstimatorService.rleType from instance method new org.apache.hadoop.resourceestimator.service.ResourceEstimatorService() At ResourceEstimatorService.java:from instance method new org.apache.hadoop.resourceestimator.service.ResourceEstimatorService() At ResourceEstimatorService.java:[line 108] Write to static field org.apache.hadoop.resourceestimator.service.ResourceEstimatorService.skylineStore from instance method new org.apache.hadoop.resourceestimator.service.ResourceEstimatorService() At ResourceEstimatorService.java:from instance method new org.apache.hadoop.resourceestimator.service.ResourceEstimatorService() At ResourceEstimatorService.java:[line 82] Write to static field org.apache.hadoop.resourceestimator.service.ResourceEstimatorService.skylineStoreType from instance method new org.apache.hadoop.resourceestimator.service.ResourceEstimatorService() At ResourceEstimatorService.java:from instance method new org.apache.hadoop.resourceestimator.service.ResourceEstimatorService() At ResourceEstimatorService.java:[line 111] Write to static field org.apache.hadoop.resourceestimator.service.ResourceEstimatorService.solver from instance method new org.apache.hadoop.resourceestimator.service.ResourceEstimatorService() At ResourceEstimatorService.java:from instance method new org.apache.hadoop.resourceestimator.service.ResourceEstimatorService() At ResourceEstimatorService.java:[line 92] Found reliance on default encoding in org.apache.hadoop.resourceestimator.translator.impl.BaseLogParser.parseStream(InputStream):in org.apache.hadoop.resourceestimator.translator.impl.BaseLogParser.parseStream(InputStream): new java.io.InputStreamReader(InputStream) At BaseLogParser.java:[line 104] org.apache.hadoop.resourceestimator.translator.impl.LogParserUtil.parseLog(String) may fail to clean up java.io.InputStream Obligation to clean up resource created at LogParserUtil.java:up java.io.InputStream Obligation to clean up resource created at LogParserUtil.java:[line 94] is not discharged FindBugs : module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager org.apache.hadoop.yarn.serv
Re: [VOTE] Merge yarn-native-services branch into trunk
Few more things: This is the document for trying a non-docker service on YARN. https://github.com/apache/hadoop/blob/yarn-native-services/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/yarn-service/QuickStart.md And the document for a docker based service https://github.com/apache/hadoop/blob/yarn-native-services/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/yarn-service/Examples.md And the vote lasts 7 days as usual. Thanks, Jian On Oct 30, 2017, at 1:06 PM, Jian He mailto:j...@hortonworks.com>> wrote: Hi All, I would like to restart the vote for merging yarn-native-services to trunk. Since last vote, we have been working on several issues in documentation, DNS, CLI modifications etc. We believe now the feature is in a much better shape. Some back ground: At a high level, the following are the key feautres implemented. - YARN-5079[1]. A native YARN framework (ApplicationMaster) to orchestrate existing services to YARN either docker or non-docker based. - YARN-4793[2]. A Rest API service embeded in RM (optional) for user to deploy a service via a simple JSON spec - YARN-4757[3]. Extending today's service registry with a simple DNS service to enable users to discover services deployed on YARN via standard DNS lookup - YARN-6419[4]. UI support for native-services on the new YARN UI All these new services are optional and are sitting outside of the existing system, and have no impact on existing system if disabled. Special thanks to a team of folks who worked hard towards this: Billie Rinaldi, Gour Saha, Vinod Kumar Vavilapalli, Jonathan Maron, Rohith Sharma K S, Sunil G, Akhil PB, Eric Yang. This effort could not be possible without their ideas and hard work. Also thanks Allen for some review and verifications. Thanks, Jian [1] https://issues.apache.org/jira/browse/YARN-5079 [2] https://issues.apache.org/jira/browse/YARN-4793 [3] https://issues.apache.org/jira/browse/YARN-4757 [4] https://issues.apache.org/jira/browse/YARN-6419
Re: Cutting branch-2.9
Hi Subru and Arun, Thanks for moving forward with 2.9 release. Is the first cut of 2.9 release supposed to be a stable version or just an alpha version? If it is supposed to be a stable version, we should run jdiff test and check for API compatibility before releasing out. Please let me know if you need any help here. Thanks, Junping From: Subru Krishnan Sent: Monday, October 30, 2017 12:39 PM To: common-dev@hadoop.apache.org; yarn-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org Cc: Arun Suresh Subject: Cutting branch-2.9 We want to give heads up that we are going to cut branch-2.9 tomorrow morning. We are down to 3 blockers and they all are close to being committed (thanks everyone): https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.9+Release There are 4 other non-blocker JIRAs that are targeted for 2.9.0 which are close to completion. Folks who are working/reviewing these, kindly prioritize accordingly so that we can make the release on time. https://issues.apache.org/jira/browse/YARN-7398?filter=12342468 Thanks in advance! -Subru/Arun - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Re: Cutting branch-2.9
Hello Jumping Thanks for chiming in.. Appreciate the offer to help. I did run jdiff on branch-2 today.. did not find any red flags. Will post the report here shortly for review. Cheers -Arun On Oct 30, 2017 4:44 PM, "Junping Du" wrote: > Hi Subru and Arun, > Thanks for moving forward with 2.9 release. Is the first cut of 2.9 > release supposed to be a stable version or just an alpha version? If it is > supposed to be a stable version, we should run jdiff test and check for API > compatibility before releasing out. Please let me know if you need any help > here. > > Thanks, > > Junping > > From: Subru Krishnan > Sent: Monday, October 30, 2017 12:39 PM > To: common-dev@hadoop.apache.org; yarn-...@hadoop.apache.org; > hdfs-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org > Cc: Arun Suresh > Subject: Cutting branch-2.9 > > We want to give heads up that we are going to cut branch-2.9 tomorrow > morning. > > We are down to 3 blockers and they all are close to being committed (thanks > everyone): > https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.9+Release > > There are 4 other non-blocker JIRAs that are targeted for 2.9.0 which are > close to completion. > > Folks who are working/reviewing these, kindly prioritize accordingly so > that we can make the release on time. > https://issues.apache.org/jira/browse/YARN-7398?filter=12342468 > > Thanks in advance! > > -Subru/Arun > > > > - > To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org > For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org > >
[jira] [Created] (HADOOP-14996) wasb: ReadFully occasionally fails when using page blobs
Thomas Marquardt created HADOOP-14996: - Summary: wasb: ReadFully occasionally fails when using page blobs Key: HADOOP-14996 URL: https://issues.apache.org/jira/browse/HADOOP-14996 Project: Hadoop Common Issue Type: Bug Components: fs/azure Reporter: Thomas Marquardt Assignee: Thomas Marquardt Looks like there is a functional bug or concurrency bug in the PageBlobInputStream implemenation of ReadFully. 1) Use 1 mapper to copy results in success: hadoop distcp -m 1 wasb://hbt-lifet...@salsbx01sparkdata.blob.core.windows.net/hive_tables wasb://hbt-lifetime-...@supporttestl2.blob.core.windows.net/hdi_backup 2) Turn on DEBUG log by setting mapreduce.map.log.level=DEBUG in ambari. Then run with more than 1 mapper: Saw debug log like this: 2017-10-27 06:18:53,545 DEBUG [main] org.apache.hadoop.fs.azure.NativeAzureFileSystem: Seek to position 136251. Bytes skipped 136210 2017-10-27 06:18:53,549 DEBUG [main] org.apache.hadoop.fs.azure.AzureNativeFileSystemStore: Closing page blob output stream. 2017-10-27 06:18:53,549 DEBUG [main] org.apache.hadoop.fs.azure.AzureNativeFileSystemStore: java.util.concurrent.ThreadPoolExecutor@73dce0e6[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0] 2017-10-27 06:18:53,549 DEBUG [main] org.apache.hadoop.security.UserGroupInformation: PrivilegedActionException as:mssupport (auth:SIMPLE) cause:java.io.EOFException 2017-10-27 06:18:53,553 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : java.io.EOFException at java.io.DataInputStream.readFully(DataInputStream.java:197) at java.io.DataInputStream.readFully(DataInputStream.java:169) at org.apache.hadoop.io.SequenceFile$Reader.sync(SequenceFile.java:2693) at org.apache.hadoop.mapreduce.lib.input.SequenceFileRecordReader.initialize(SequenceFileRecordReader.java:58) at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:548) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:786) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:170) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:164) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Re: Hadoop Compatability Guide, Part Deux: Developer Docs
We've now gone a couple of rounds of reviews on HADOOP-14876, and a patch is posted for HADOOP-14875. Feedback is very welcome. Please take a look. Daniel On 10/14/17 8:46 AM, Daniel Templeton wrote: I just posted a first patch for HADOOP-14876 that adds downstream developer docs based on the overhauled compatibility guide from HADOOP-13714. I would really appreciate some critical review of the doc, as it's much more likely to be read by downstream developers than the compatibility spec itself. There's another doc coming in HADOOP-14875 that will add what amounts to upgrade docs for admins. When that's complete, I will send another email here to solicit reviews. Daniel - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14997) AliyunOSS: follow-ups to HADOOP-13687
Genmao Yu created HADOOP-14997: -- Summary: AliyunOSS: follow-ups to HADOOP-13687 Key: HADOOP-14997 URL: https://issues.apache.org/jira/browse/HADOOP-14997 Project: Hadoop Common Issue Type: Sub-task Components: fs/oss Reporter: Genmao Yu Assignee: Genmao Yu Priority: Minor -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Re: Access to Confluence Wiki
Done. Welcome! -Akira On 2017/10/28 3:26, Hanisha Koneru wrote: Hi, Can I please get access to the Confluence Hadoop Wiki. My confluence id is “hanishakoneru”. Thanks, Hanisha - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org