Re: [DISCUSS] Release numbering semantics with concurrent (>2) releases [Was Setting JIRA fix versions for 3.0.0 releases]
> I think I understand a bit better, though now I ask how this date is > different from the release date. OIC. I also assume that the freezing branch cannot include the changes between freezing date and the release date. This is for strict ordering to ensure which is the newer. If we have lots maintenance branches, it helps us to understand which branches include fix a problem of my cluster. > I think this problem is also pretty rare in practice, since users normally > upgrade to the highest maintenance release within a major/minor. If there will be lots maintenance branches in parallel(2.6.x, 2.7.x, 2.8.x, 2.9.x, 3.0.x), we can hit this problem more easily - if a user uses plan to upgrade 2.7.3 cluster to 2.8.3 or 2.9.1 or 3.0.1, which version should the user choose? This is my concern. However, as you mentioned, we decide to reduce the number of branches we keep maintenance, we don't need to do that. Best, - Tsuyoshi On Wed, Jul 27, 2016 at 3:40 PM, Andrew Wang wrote: > I think I understand a bit better, though now I ask how this date is > different from the release date. Based on the HowToRelease instructions, we > set the release date to when the release vote passes. So, start of release > vote vs. end of release vote doesn't seem that different, and these dates > are still totally ordered. > > For the user in this scenario, she can upgrade from 2.7.3 to any later 2.7.c > release (found easily since a.b.c releases are ordered), and when jumping to > a new minor or major version, any version released chronologically after > 2.7.3. This means you need to check the website, but given that this is the > way most enterprise software is versioned, I think it'll be okay by users. > > I think this problem is also pretty rare in practice, since users normally > upgrade to the highest maintenance release within a major/minor. Thus > they'll only hit this if their upgrade cycle is faster than it takes for a > change released in e.g. 2.6.x to then also be released in a 2.7.x. > > Best, > Andrew > > On Tue, Jul 26, 2016 at 11:13 PM, Tsuyoshi Ozawa wrote: >> >> > Andrew: I bet many would assume it's the release date, like how Ubuntu >> > releases are numbered. >> >> Good point. Maybe I confuse you because of lack of explanation. >> >> I assume that "branch-cut off timing" mean the timing of freezing branch >> like when starting the release vote. It's because that the release can be >> delayed after the release pass. Does it make sense to you? >> >> > Even if we have the branch-cut date in the version string, devs still >> > need to be aware of other branches and backport appropriately. >> >> Yes, you're right. The good point of including date is that we can declare >> which version includes the latest changes. It helps users, not devs >> basically, to decide which version users will use: e.g. if 2.8.1-20160801 is >> released after 2.9.0-20160701 and a user uses 2.7.3-20160701, she can update >> their cluster 2.8.1, which include bug fixes against 2.7.3. Please let me >> know if I have some missing points. >> >> Thanks, >> - Tsuyoshi >> >> On Wednesday, 27 July 2016, Andrew Wang wrote: >>> >>> Thanks for replies Akira and Tsuyoshi, inline: >>> Akira: Assuming 3.0.0-alpha1 will be released between 2.7.0 and 2.8.0, we need to add 3.0.0-alphaX if 2.8.0 is in the fix versions of a jira and we don't need to add 3.0.0-alphaX if 2.7.0 is in the fix versions of a jira. Is it right? >>> >>> >>> Yes, correct. >>> Tsuyoshi: My suggestion is adding the date when branch cut is done: like 3.0.0-alpha1-20160724, 2.8.0-20160730 or something. Pros:-) It's totally ordered. If we have a policy such as backporting to maintainance branches after the date, users can find that which version is cutting edge. In the example of above, 2.8.0-20160730 can include bug fixes which is not included in 3.0.0-alpha1-20160724. Cons:-( A bit redundant. >>> Could you elaborate on the problem this scheme addresses? We always want >>> our releases, when ordered chronologically, to incorporate all the known >>> relevant bug fixes. Even if we have the branch-cut date in the version >>> string, devs still need to be aware of other branches and backport >>> appropriately. >>> >>> Given that branch cuts and releases might not happen in the same order >>> (e.g. if 3.0.0-alpha1 precedes 2.8.0), I think this also would be confusing >>> for users. I bet many would assume it's the release date, like how Ubuntu >>> releases are numbered. >>> >>> Best, >>> Andrew > > - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: [DISCUSS] Release numbering semantics with concurrent (>2) releases [Was Setting JIRA fix versions for 3.0.0 releases]
Thanks Andrew for sharing your thoughts, It looks better if we can put multiple versions on the fix version, with that we can at least do some queries on JIRA to check the issues like "in branch-2.6.5 but not in branch-2.7.4". I still have a couple of questions: *1) How CHANGES.txt (or release note) should look like?* Not sure if this is discussed before. Previously we put the *next* release version of earliest branch to CHANGES.txt. However, this could be confusing and need a lot of manual works. For example, we have two parallel release branches: branch-2.6.5 and branch-2.7.4. When we need to backport a commit X in branch-2.7.4 to branch-2.6.5, we will update CHANGES.txt in branch-2.7.4 to say this commit X is included by Hadoop-2.6.5. However, if we release Hadoop-2.7.4 before Hadoop-2.6.5, user will find the Hadoop-2.6.5 is not released yet. To me, we should put the fix version in CHANGES.txt to the released Hadoop from the earliest branch, in the above example, Hadoop-2.7.4 should be the fix version of commit X in release note of Hadoop-2.7.4. Instead, I suggest to add a suffix ("released") to the fix version after release is done. So the release note generator can do query easier, and other user of JIRA can benefit from this to understand which releases include a given JIRA. *2) Do we need to update historical JIRAs?* It's better to make a consistent rule for active release branches (to me they're branch-2.6 and up). So it will be better to update fix version for all resolved JIRAs in release branches. Thoughts? Wangda On Tue, Jul 26, 2016 at 11:40 PM, Andrew Wang wrote: > I think I understand a bit better, though now I ask how this date is > different from the release date. Based on the HowToRelease instructions, we > set the release date to when the release vote passes. So, start of release > vote vs. end of release vote doesn't seem that different, and these dates > are still totally ordered. > > For the user in this scenario, she can upgrade from 2.7.3 to any later > 2.7.c release (found easily since a.b.c releases are ordered), and when > jumping to a new minor or major version, any version released > chronologically after 2.7.3. This means you need to check the website, but > given that this is the way most enterprise software is versioned, I think > it'll be okay by users. > > I think this problem is also pretty rare in practice, since users normally > upgrade to the highest maintenance release within a major/minor. Thus > they'll only hit this if their upgrade cycle is faster than it takes for a > change released in e.g. 2.6.x to then also be released in a 2.7.x. > > Best, > Andrew > > On Tue, Jul 26, 2016 at 11:13 PM, Tsuyoshi Ozawa wrote: > > > > Andrew: I bet many would assume it's the release date, like how Ubuntu > > releases are numbered. > > > > Good point. Maybe I confuse you because of lack of explanation. > > > > I assume that "branch-cut off timing" mean the timing of freezing branch > > like when starting the release vote. It's because that the release can > > be delayed after the release pass. Does it make sense to you? > > > > > Even if we have the branch-cut date in the version string, devs still > > need to be aware of other branches and backport appropriately. > > > > Yes, you're right. The good point of including date is that we can > declare > > which version includes the latest changes. It helps users, not devs > > basically, to decide which version users will use: e.g. if > > 2.8.1-20160801 is released after 2.9.0-20160701 and a user uses > > 2.7.3-20160701, she can update their cluster 2.8.1, which include bug > fixes > > against 2.7.3. Please let me know if I have some missing points. > > > > Thanks, > > - Tsuyoshi > > > > On Wednesday, 27 July 2016, Andrew Wang > wrote: > > > >> Thanks for replies Akira and Tsuyoshi, inline: > >> > >> Akira: Assuming 3.0.0-alpha1 will be released between 2.7.0 and 2.8.0, > we > >>> need to add 3.0.0-alphaX if 2.8.0 is in the fix versions of a jira and > we > >>> don't need to add 3.0.0-alphaX if 2.7.0 is in the fix versions of a > jira. > >>> Is it right? > >> > >> > >> Yes, correct. > >> > >> > >>> Tsuyoshi: My suggestion is adding the date when branch cut is done: > like > >>> 3.0.0-alpha1-20160724, 2.8.0-20160730 or something. > >>> > >>> Pros:-) It's totally ordered. If we have a policy such as backporting > >>> to maintainance branches after the date, users can find that which > >>> version > >>> is cutting edge. In the example of above, 2.8.0-20160730 can include > bug > >>> fixes which is not included in 3.0.0-alpha1-20160724. > >>> > >>> Cons:-( A bit redundant. > >>> > >>> Could you elaborate on the problem this scheme addresses? We always > want > >> our releases, when ordered chronologically, to incorporate all the known > >> relevant bug fixes. Even if we have the branch-cut date in the version > >> string, devs still need to be aware of other branches and backport > >> appropriately. > >> > >> Given that br
[jira] [Created] (HDFS-10696) TestHDFSCLI fails
Akira Ajisaka created HDFS-10696: Summary: TestHDFSCLI fails Key: HDFS-10696 URL: https://issues.apache.org/jira/browse/HDFS-10696 Project: Hadoop HDFS Issue Type: Bug Components: test Reporter: Akira Ajisaka TestHDFSCLI fails. {noformat}2016-07-27 19:53:20,790 [main] INFO cli.CLITestHelper (CLITestHelper.java:displayResults(177)) - Comparator: [RegexpComparator] 2016-07-27 19:53:20,790 [main] INFO cli.CLITestHelper (CLITestHelper.java:displayResults(179)) - Comparision result: [fail] 2016-07-27 19:53:20,791 [main] INFO cli.CLITestHelper (CLITestHelper.java:displayResults(181)) - Expected output: [^( |\t)*The storage type specific quota is cleared when -storageType option is specified.( )*] 2016-07-27 19:53:20,791 [main] INFO cli.CLITestHelper (CLITestHelper.java:displayResults(183)) - Actual output: [-clrSpaceQuota [-storageType ] ...: Clear the space quota for each directory . For each directory, attempt to clear the quota. An error will be reported if 1. the directory does not exist or is a file, or 2. user is not an administrator. It does not fault if the directory has no quota. The storage type specific quota is cleared when -storageType option is specified. Available storageTypes are - RAM_DISK - DISK - SSD - ARCHIVE ] {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-10697) Erasure Code(6:3) can not work, using get fs shell when 2 datanode down
Alberic Liu created HDFS-10697: -- Summary: Erasure Code(6:3) can not work, using get fs shell when 2 datanode down Key: HDFS-10697 URL: https://issues.apache.org/jira/browse/HDFS-10697 Project: Hadoop HDFS Issue Type: Bug Components: erasure-coding Affects Versions: 3.0.0-alpha2 Environment: 9 datanodes 2 HA name node and Yarn Reporter: Alberic Liu I can put file into EC folder(6:3), and get the file successfully. But when I shutdown one datanode, I still can get the file, using hdfs dfs -get. But when I shutdown two datanodes, when i use hdfs dfs -get, I get following error: get: 4 missing blocks, the stripe is: Offset=786432, length=65536, fetchedChunksNum=5, missingChunksNum=4; locatedBlocks is: LocatedBlocks{ -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: [DISCUSS] Release numbering semantics with concurrent (>2) releases [Was Setting JIRA fix versions for 3.0.0 releases]
Inline. > 1) Set the fix version for all a.b.c versions, where c > 0. > 2) For each major release line, set the lowest a.b.0 version. > Sounds reasonable. > > The -alphaX versions we're using leading up to 3.0.0 GA can be treated as > a.b.c versions, with alpha1 being the a.b.0 release. > Once 3.0.0 GA goes out, a user would want to see the diff from the latest 2.x.0 release (say 2.9.0). Are you suggesting 3.0.0 GA would have c = 5 (say) and hence rule 1 would apply, and it should show up in the release notes? > > As an example, if a JIRA was committed to branch-2.6, branch-2.7, branch-2, > branch-3.0.0-alpha1, and trunk, it could have fix versions of 2.6.5, 2.7.3, > 2.8.0, 3.0.0-alpha1. The first two fix versions come from application of > rule 1, and the last two fix versions come from rule 2. > > I'm very eager to move this discussion forward, so feel free to reach out > on or off list if I can help with anything. > I think it is good practice to set multiple fix versions. However, it might take the committers a little bit to learn. Since the plan is to cut 3.0.0 off trunk, can we just bulk edit to add the 3.0.0-alphaX version? > > Best, > Andrew >
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/115/ [Jul 26, 2016 1:30:02 PM] (stevel) Revert "HDFS-10668. Fix intermittently failing UT [Jul 26, 2016 1:53:37 PM] (kai.zheng) HADOOP-13041. Adding tests for coder utilities. Contributed by Kai [Jul 26, 2016 3:01:42 PM] (weichiu) HDFS-9937. Update dfsadmin command line help and HdfsQuotaAdminGuide. [Jul 26, 2016 3:19:06 PM] (varunsaxena) YARN-5431. TimelineReader daemon start should allow to pass its own [Jul 26, 2016 3:43:12 PM] (varunsaxena) Revert "YARN-5431. TimelineReader daemon start should allow to pass its [Jul 26, 2016 7:27:46 PM] (arp) HDFS-10642. [Jul 26, 2016 9:54:03 PM] (Arun Suresh) YARN-5392. Replace use of Priority in the Scheduling infrastructure with [Jul 26, 2016 10:33:20 PM] (cnauroth) HADOOP-13422. ZKDelegationTokenSecretManager JaasConfig does not work [Jul 26, 2016 11:01:50 PM] (weichiu) HDFS-10598. DiskBalancer does not execute multi-steps plan. Contributed [Jul 27, 2016 1:14:09 AM] (wangda) YARN-5342. Improve non-exclusive node partition resource allocation in [Jul 27, 2016 2:08:30 AM] (Arun Suresh) YARN-5351. ResourceRequest should take ExecutionType into account during [Jul 27, 2016 4:22:59 AM] (wangda) YARN-5195. RM intermittently crashed with NPE while handling [Jul 27, 2016 4:56:42 AM] (brahma) HDFS-10668. Fix intermittently failing UT -1 overall The following subsystems voted -1: asflicense unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.cli.TestHDFSCLI hadoop.yarn.client.api.impl.TestYarnClient hadoop.yarn.server.nodemanager.TestDirectoryCollection hadoop.yarn.server.resourcemanager.scheduler.TestSchedulerHealth hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler hadoop.yarn.server.resourcemanager.scheduler.fifo.TestFifoScheduler hadoop.yarn.server.resourcemanager.TestResourceManager hadoop.yarn.server.TestMiniYarnClusterNodeUtilization hadoop.yarn.server.TestContainerManagerSecurity cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/115/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/115/artifact/out/diff-compile-javac-root.txt [172K] checkstyle: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/115/artifact/out/diff-checkstyle-root.txt [16M] pylint: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/115/artifact/out/diff-patch-pylint.txt [16K] shellcheck: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/115/artifact/out/diff-patch-shellcheck.txt [20K] shelldocs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/115/artifact/out/diff-patch-shelldocs.txt [16K] whitespace: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/115/artifact/out/whitespace-eol.txt [12M] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/115/artifact/out/whitespace-tabs.txt [1.3M] javadoc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/115/artifact/out/diff-javadoc-javadoc-root.txt [2.3M] unit: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/115/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [144K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/115/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-nativetask.txt [124K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/115/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt [16K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/115/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt [36K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/115/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt [64K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/115/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt [268K] asflicense: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/115/artifact/out/patch-asflicense-problems.txt [4.0K] Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev
Re: [DISCUSS] Release numbering semantics with concurrent (>2) releases [Was Setting JIRA fix versions for 3.0.0 releases]
Thanks Vinod for bringing up this topic for discussion. I share the same concern here from my previous experience and I doubt some simple rules proposed below could make life easier. > The question now is what we do for the 2.8.0 and 3.0.0-alpha1 fix versions. > Allen's historical perspective is that we've based each minor or major > release off of the previous minor release. So, 2.8.0 would be based off of > 2.7.0. Assuming 3.0.0-alpha1 happens before 2.8.0, 3.0.0-alpha1 would also > be based off of 2.7.0. This also makes sense from a user POV; someone on a > 2.6.x going to 3.0.0-alpha1 can look at the 2.7.0 and 3.0.0-alpha1 notes to > see what's changed. This is not correct - not reflecting the past and not helpful for the future. There is no benefit to claim 3.0.0-alpha1 is based on 2.7.0 over 2.7.3 (in case 2.8.0 is not there). In the past, for example, when we cut off 2.7, we already have 2.6.0 and 2.6.1 get released, so 2.7.0 take all commits from 2.6.1 (not 2.6.0). In future, assume when we start the release effor of 3.1.0 and we have 3.0.1, 3.0.2, etc., 3.0.x should be more stable than 3.0.0-alpha, so there is unnecessary to do everything from scratch (3.0.0-alpha). So the rule here should be: a new major or minor release should come from a release: 1. tag with stable 2. released latest 3. with maximum version number If condition 2 and 3 get conflicts, we should give priority to 3. For example, when 3.0.0-alpha1 is about to release, assume we have 2.8.0, 2.7.4 and 2.7.4 get released after 2.8.0, then we should claim 3.0.0-alpha is based on 2.8.0 instead of 2.7.4. > As an example, if a JIRA was committed to branch-2.6, branch-2.7, branch-2, > branch-3.0.0-alpha1, and trunk, it could have fix versions of 2.6.5, 2.7.3, > 2.8.0, 3.0.0-alpha1. The first two fix versions come from application of > rule 1, and the last two fix versions come from rule 2. I don't think setting version tags to be more than 3 is a good practice. The example above means we need to backport this patch to 5 branches which make our committers' life really tough - it requires more effort of committing a patch and also increases the risky of bugs that caused by backport. Given realistic community review bandwidth (please check another thread from Chris D.), I strongly suggest we keep active release train to be less than 3, so we can have 2 stable release or 1 stable release + 1 alpha release in releasing. BTW, I never see we have a clear definition for alpha release. It is previously used as unstable in API definition (2.1-alpha, 2.2-alpha, etc.) but sometimes means unstable in production quality (2.7.0). I think we should clearly define it with major consensus so user won't misunderstanding the risky here. Also, if we treat our 3.0.0-alpha release work seriously, we should also think about trunk's version number issue (bump up to 4.0.0-alpha?) or there could be no room for 3.0 incompatible feature/bits soon. Just 2 cents. Thanks, Junping 2016-07-27 15:34 GMT+01:00 Karthik Kambatla : > Inline. > > > 1) Set the fix version for all a.b.c versions, where c > 0. > > 2) For each major release line, set the lowest a.b.0 version. > > > > Sounds reasonable. > > > > > > The -alphaX versions we're using leading up to 3.0.0 GA can be treated as > > a.b.c versions, with alpha1 being the a.b.0 release. > > > > Once 3.0.0 GA goes out, a user would want to see the diff from the latest > 2.x.0 release (say 2.9.0). > > Are you suggesting 3.0.0 GA would have c = 5 (say) and hence rule 1 would > apply, and it should show up in the release notes? > > > > > > As an example, if a JIRA was committed to branch-2.6, branch-2.7, > branch-2, > > branch-3.0.0-alpha1, and trunk, it could have fix versions of 2.6.5, > 2.7.3, > > 2.8.0, 3.0.0-alpha1. The first two fix versions come from application of > > rule 1, and the last two fix versions come from rule 2. > > > > I'm very eager to move this discussion forward, so feel free to reach out > > on or off list if I can help with anything. > > > > > I think it is good practice to set multiple fix versions. However, it might > take the committers a little bit to learn. > > Since the plan is to cut 3.0.0 off trunk, can we just bulk edit to add the > 3.0.0-alphaX version? > > > > > > Best, > > Andrew > > >
[jira] [Reopened] (HDFS-8224) Any IOException in DataTransfer#run() will run diskError thread even if it is not disk error
[ https://issues.apache.org/jira/browse/HDFS-8224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rushabh S Shah reopened HDFS-8224: -- Saw one more occurrence of this bug. We should atleast add this block to the front of the scanning queue. > Any IOException in DataTransfer#run() will run diskError thread even if it is > not disk error > > > Key: HDFS-8224 > URL: https://issues.apache.org/jira/browse/HDFS-8224 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.6.0 >Reporter: Rushabh S Shah >Assignee: Rushabh S Shah > Fix For: 2.8.0 > > > This happened in our 2.6 cluster. > One of the block and its metadata file were corrupted. > The disk was healthy in this case. > Only the block was corrupt. > Namenode tried to copy that block to another datanode but failed with the > following stack trace: > 2015-04-20 01:04:04,421 > [org.apache.hadoop.hdfs.server.datanode.DataNode$DataTransfer@11319bc4] WARN > datanode.DataNode: DatanodeRegistration(a.b.c.d, > datanodeUuid=e8c5135c-9b9f-4d05-a59d-e5525518aca7, infoPort=1006, > infoSecurePort=0, ipcPort=8020, > storageInfo=lv=-56;cid=CID-e7f736ac-158e-446e-9091-7e66f3cddf3c;nsid=358250775;c=1428471998571):Failed > to transfer BP-xxx-1351096255769:blk_2697560713_1107108863999 to > a1.b1.c1.d1:1004 got > java.io.IOException: Could not create DataChecksum of type 0 with > bytesPerChecksum 0 > at > org.apache.hadoop.util.DataChecksum.newDataChecksum(DataChecksum.java:125) > at > org.apache.hadoop.hdfs.server.datanode.BlockMetadataHeader.readHeader(BlockMetadataHeader.java:175) > at > org.apache.hadoop.hdfs.server.datanode.BlockMetadataHeader.readHeader(BlockMetadataHeader.java:140) > at > org.apache.hadoop.hdfs.server.datanode.BlockMetadataHeader.readDataChecksum(BlockMetadataHeader.java:102) > at > org.apache.hadoop.hdfs.server.datanode.BlockSender.(BlockSender.java:287) > at > org.apache.hadoop.hdfs.server.datanode.DataNode$DataTransfer.run(DataNode.java:1989) > at java.lang.Thread.run(Thread.java:722) > The following catch block in DataTransfer#run method will treat every > IOException as disk error fault and run disk errror > {noformat} > catch (IOException ie) { > LOG.warn(bpReg + ":Failed to transfer " + b + " to " + > targets[0] + " got ", ie); > // check if there are any disk problem > checkDiskErrorAsync(); > } > {noformat} > This block was never scanned by BlockPoolSliceScanner otherwise it would have > reported as corrupt block. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: Improving recovery performance for degraded reads
Hi Roy, (a) In your last email, I am sure you meant => "... submitting read requests to fetch "any" (instead of all) the 'k' chunk (out of k+m-x surviving chunks) ? Do you have any optimization in place to decide which data-nodes will be part of those "k" ? Answer:- I hope you know the write path, just adding few details here to support the read explanation part. While writing to an EC file, dfs client writes data stripe(e.g. 64KB cellsize) to multiple datanodes. For (k, m) schema, the client writes data block to the first k datanodes and parity block to the remaining m datanodes. Say, one stripe is (k * cellSize + m * cellSize) data. While reading, client will fetch in the same order, read stripe by stripe. The datanodes with data blocks are first fetched than the datanodes with parity blocks because less EC block reconstruction work is needed. Internally, dfs client reads the whole stripe one by one and contacts k datanodes parallelly for each stripe. If there is any failures then will contact parity datanodes and do reconstruction on the fly. 'DFSStripedInputStream' supported both positional read and read entire buffer(e.g filesize buffer). (b) Are there any caching being done (as proposed for QFS in the previously attached "PPR" paper) ? Answer:- HDFS-9879, there is an open jira to discuss the caching of striped blocks at the datanode. Perhaps, caching logic could be utilized similar to the QFS and while reconstruction choose those datanodes that have already cached the data in memory. This is an open improvement task as of now. (c) When you mentioned stripping is being done, I assume it is probably to reduce the chunk sizes and hence k*c ? Answer:- Yes, striping is done by dividing the block into several chunks, we call it as cellSize (e.g. 64KB). (k * c + m * c) is one stripe. A block group comprises of several stripes. I'd suggest you to read the blog - http://blog.cloudera.com/blog/2015/09/introduction-to-hdfs-erasure-coding-in-apache-hadoop/ to understand more about the stripe, cells and block group terminolgy etc before reeading the below answer. blk_0 blk_1 blk_2 | || vv v +--+ +--+ +--+ |cell_0| |cell_1| |cell_2| +--+ +--+ +--+ Now, if my object sizes are large (e.g. super HD images) where I would have to get data from multiple stripes to rebuild the images before I can display to the client, do you think stripping would still help ? Is there a possibility that since I know that all the segments of the HD image would always be read together, by stripping and distributing it on different nodes, I am ignoring its special/temporal locality and further increase any associated delays ? Answer:- Since for each stripe it contacts all the k datanodes, assume if there are slow datanodes or some dead datanodes in each data block stripe then it will affect the read performance. AFAIK, for a large file contiguous layout is suitable, this will be supported in phase-2 and design discussions are still going on, please see HDFS-8030 jira. On the otherside, in theory I can say there is a benefit of striping layout, which enables the client to work with multiple data nodes in parallel, greatly enhancing the aggregate throughput(assuming that all datanodes are good servers). But this needs to be tested in your cluster to understand the impact. Thanks, Rakesh Intel On Sun, Jul 24, 2016 at 12:00 PM, Roy Leonard wrote: > Hi Rakesh, > > Thanks for sharing your thoughts and updates. > > (a) In your last email, I am sure you meant => "... submitting read > requests to fetch "any" (instead of all) the 'k' chunk (out of k+m-x > surviving chunks) ? > Do you have any optimization in place to decide which data-nodes will be > part of those "k" ? > > (b) Are there any caching being done (as proposed for QFS in the previously > attached "PPR" paper) ? > > (c) When you mentioned stripping is being done, I assume it is probably to > reduce the chunk sizes and hence k*c ? > Now, if my object sizes are large (e.g. super HD images) where I would have > to get data from multiple stripes to rebuild the images before I can > display to the client, do you think stripping would still help ? > Is there a possibility that since I know that all the segments of the HD > image would always be read together, by stripping and distributing it on > different nodes, I am ignoring its special/temporal locality and further > increase any associated delays ? > > Just wanted to know your thoughts. > I am looking forward to the future performance improvements in HDFS. > > Regards, > R. > > On Fri, Jul 22, 2016 at 8:52 AM, Rakesh Radhakrishnan > wrote: > > > I'm adding one more point to the above. In my previous mail reply, I've > > explained the striped block reconstruction task which will be triggered > by > > the Namenode on identifying a missing/bad block. Similarly, in case
Re: [VOTE] Release Apache Hadoop 2.7.3 RC0
+1 (binding) - Downloaded binary tarball - verified signatures - setup pseudo cluster - ran some of the example jobs, clicked around the UI a bit - Robert On Mon, Jul 25, 2016 at 3:28 PM, Jason Lowe wrote: > +1 (binding) > - Verified signatures and digests- Built from source with native support- > Deployed a pseudo-distributed cluster- Ran some sample jobs > Jason > > From: Vinod Kumar Vavilapalli > To: "common-...@hadoop.apache.org" ; > hdfs-dev@hadoop.apache.org; yarn-...@hadoop.apache.org; " > mapreduce-...@hadoop.apache.org" > Cc: Vinod Kumar Vavilapalli > Sent: Friday, July 22, 2016 9:15 PM > Subject: [VOTE] Release Apache Hadoop 2.7.3 RC0 > > Hi all, > > I've created a release candidate RC0 for Apache Hadoop 2.7.3. > > As discussed before, this is the next maintenance release to follow up > 2.7.2. > > The RC is available for validation at: > http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/ < > http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/> > > The RC tag in git is: release-2.7.3-RC0 > > The maven artifacts are available via repository.apache.org < > http://repository.apache.org/> at > https://repository.apache.org/content/repositories/orgapachehadoop-1040/ < > https://repository.apache.org/content/repositories/orgapachehadoop-1040/> > > The release-notes are inside the tar-balls at location > hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I > hosted this at > http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/releasenotes.html < > http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html> for > your quick perusal. > > As you may have noted, a very long fix-cycle for the License & Notice > issues (HADOOP-12893) caused 2.7.3 (along with every other Hadoop release) > to slip by quite a bit. This release's related discussion thread is linked > below: [1]. > > Please try the release and vote; the vote will run for the usual 5 days. > > Thanks, > Vinod > > [1]: 2.7.3 release plan: > https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html < > http://markmail.org/thread/6yv2fyrs4jlepmmr> > > >
Re: [DISCUSS] Retire BKJM from trunk?
I recommend including the BookKeeper community in this discussion. I’ve added their user@ and dev@ lists to this thread. I do not see BKJM being used in practice. Removing it from trunk would be attractive in terms of less code for Hadoop to maintain and build, but if we find existing users that want to keep it, I wouldn’t object. --Chris Nauroth On 7/26/16, 11:14 PM, "Vinayakumar B" wrote: Hi All, BKJM was Active and made much stable when the NameNode HA was implemented and there was no QJM implemented. Now QJM is present and is much stable which is adopted by many production environment. I wonder whether it would be a good time to retire BKJM from trunk? Are there any users of BKJM exists? -Vinay - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: [DISCUSS] Release numbering semantics with concurrent (>2) releases [Was Setting JIRA fix versions for 3.0.0 releases]
> > The -alphaX versions we're using leading up to 3.0.0 GA can be treated as >> a.b.c versions, with alpha1 being the a.b.0 release. >> > > Once 3.0.0 GA goes out, a user would want to see the diff from the latest > 2.x.0 release (say 2.9.0). > > Are you suggesting 3.0.0 GA would have c = 5 (say) and hence rule 1 would > apply, and it should show up in the release notes? > > Yea. It means the if you're coming from 2.7, you'd read the 3.0.0-alphas, 3.0.0-betas, and finally the 3.0.0 GA notes. The website notes will aggregate all the alpha and beta changes leading up to 3.0.0 GA, so is likely where users will turn to first. > >> As an example, if a JIRA was committed to branch-2.6, branch-2.7, >> branch-2, >> branch-3.0.0-alpha1, and trunk, it could have fix versions of 2.6.5, >> 2.7.3, >> 2.8.0, 3.0.0-alpha1. The first two fix versions come from application of >> rule 1, and the last two fix versions come from rule 2. >> >> I'm very eager to move this discussion forward, so feel free to reach out >> on or off list if I can help with anything. >> > > > I think it is good practice to set multiple fix versions. However, it > might take the committers a little bit to learn. > > Since the plan is to cut 3.0.0 off trunk, can we just bulk edit to add the > 3.0.0-alphaX version? > > Yea, I have a script and some JIRA queries that can help with this. I'll also plan to compare with git log contents for extra verification.
[jira] [Created] (HDFS-10698) Test org.apache.hadoop.cli.TestHDFSCLI fails in trunk
Yongjun Zhang created HDFS-10698: Summary: Test org.apache.hadoop.cli.TestHDFSCLI fails in trunk Key: HDFS-10698 URL: https://issues.apache.org/jira/browse/HDFS-10698 Project: Hadoop HDFS Issue Type: Bug Components: hdfs Reporter: Yongjun Zhang {code} Running org.apache.hadoop.cli.TestHDFSCLI Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 39.887 sec <<< FAILURE! - in org.apache.hadoop.cli.TestHDFSCLI testAll(org.apache.hadoop.cli.TestHDFSCLI) Time elapsed: 39.697 sec <<< FAILURE! java.lang.AssertionError: One of the tests failed. See the Detailed results to identify the command that failed at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.assertTrue(Assert.java:41) at org.apache.hadoop.cli.CLITestHelper.displayResults(CLITestHelper.java:263) at org.apache.hadoop.cli.CLITestHelper.tearDown(CLITestHelper.java:125) at org.apache.hadoop.cli.TestHDFSCLI.tearDown(TestHDFSCLI.java:87) Results : Failed tests: TestHDFSCLI.tearDown:87->CLITestHelper.tearDown:125->CLITestHelper.displayResults:263 One of the tests failed. See the Detailed results to identify the command that failed Tests run: 1, Failures: 1, Errors: 0, Skipped: 0 {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 2.7.3 RC0
+1 (non-binding) - Build source with Java 1.8.0_101 on Centos 7.2 with native - Build source with Java 1.7.0_79 on Mac - Verify license and notice using the shell script in HADOOP-13374 - Deploy a pseudo cluster - Run basic dfs, distcp, ACL, webhdfs commands - Run MapReduce workcount and pi examples - Start balancer Thanks, John John Zhuge Software Engineer, Cloudera On Wed, Jul 27, 2016 at 11:38 AM, Robert Kanter wrote: > +1 (binding) > > - Downloaded binary tarball > - verified signatures > - setup pseudo cluster > - ran some of the example jobs, clicked around the UI a bit > > > - Robert > > > On Mon, Jul 25, 2016 at 3:28 PM, Jason Lowe > wrote: > > > +1 (binding) > > - Verified signatures and digests- Built from source with native support- > > Deployed a pseudo-distributed cluster- Ran some sample jobs > > Jason > > > > From: Vinod Kumar Vavilapalli > > To: "common-...@hadoop.apache.org" ; > > hdfs-dev@hadoop.apache.org; yarn-...@hadoop.apache.org; " > > mapreduce-...@hadoop.apache.org" > > Cc: Vinod Kumar Vavilapalli > > Sent: Friday, July 22, 2016 9:15 PM > > Subject: [VOTE] Release Apache Hadoop 2.7.3 RC0 > > > > Hi all, > > > > I've created a release candidate RC0 for Apache Hadoop 2.7.3. > > > > As discussed before, this is the next maintenance release to follow up > > 2.7.2. > > > > The RC is available for validation at: > > http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/ < > > http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/> > > > > The RC tag in git is: release-2.7.3-RC0 > > > > The maven artifacts are available via repository.apache.org < > > http://repository.apache.org/> at > > https://repository.apache.org/content/repositories/orgapachehadoop-1040/ > < > > https://repository.apache.org/content/repositories/orgapachehadoop-1040/ > > > > > > The release-notes are inside the tar-balls at location > > hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I > > hosted this at > > http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/releasenotes.html < > > http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html> > for > > your quick perusal. > > > > As you may have noted, a very long fix-cycle for the License & Notice > > issues (HADOOP-12893) caused 2.7.3 (along with every other Hadoop > release) > > to slip by quite a bit. This release's related discussion thread is > linked > > below: [1]. > > > > Please try the release and vote; the vote will run for the usual 5 days. > > > > Thanks, > > Vinod > > > > [1]: 2.7.3 release plan: > > https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html > < > > http://markmail.org/thread/6yv2fyrs4jlepmmr> > > > > > > >
[jira] [Resolved] (HDFS-10698) Test org.apache.hadoop.cli.TestHDFSCLI fails in trunk
[ https://issues.apache.org/jira/browse/HDFS-10698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HDFS-10698. Resolution: Duplicate thx Youngjun for filing the jira. This is a dup of HDFS-10696 where a patch is provided. > Test org.apache.hadoop.cli.TestHDFSCLI fails in trunk > - > > Key: HDFS-10698 > URL: https://issues.apache.org/jira/browse/HDFS-10698 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Reporter: Yongjun Zhang > > {code} > Running org.apache.hadoop.cli.TestHDFSCLI > Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 39.887 sec > <<< FAILURE! - in org.apache.hadoop.cli.TestHDFSCLI > testAll(org.apache.hadoop.cli.TestHDFSCLI) Time elapsed: 39.697 sec <<< > FAILURE! > java.lang.AssertionError: One of the tests failed. See the Detailed results > to identify the command that failed > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.assertTrue(Assert.java:41) > at > org.apache.hadoop.cli.CLITestHelper.displayResults(CLITestHelper.java:263) > at > org.apache.hadoop.cli.CLITestHelper.tearDown(CLITestHelper.java:125) > at org.apache.hadoop.cli.TestHDFSCLI.tearDown(TestHDFSCLI.java:87) > Results : > Failed tests: > > TestHDFSCLI.tearDown:87->CLITestHelper.tearDown:125->CLITestHelper.displayResults:263 > One of the tests failed. See the Detailed results to identify the command > that failed > Tests run: 1, Failures: 1, Errors: 0, Skipped: 0 > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: [DISCUSS] Retire BKJM from trunk?
+ Rakesh and Uma Rakesh and Uma might have a better idea on this. I think Huawei was using it when Rakesh and Uma worked there. - Sijie On Wed, Jul 27, 2016 at 12:06 PM, Chris Nauroth wrote: > I recommend including the BookKeeper community in this discussion. I’ve > added their user@ and dev@ lists to this thread. > > I do not see BKJM being used in practice. Removing it from trunk would be > attractive in terms of less code for Hadoop to maintain and build, but if > we find existing users that want to keep it, I wouldn’t object. > > --Chris Nauroth > > On 7/26/16, 11:14 PM, "Vinayakumar B" wrote: > > Hi All, > >BKJM was Active and made much stable when the NameNode HA was > implemented and there was no QJM implemented. >Now QJM is present and is much stable which is adopted by many > production environment. >I wonder whether it would be a good time to retire BKJM from trunk? > >Are there any users of BKJM exists? > > -Vinay > > >
Re: [DISCUSS] Release numbering semantics with concurrent (>2) releases [Was Setting JIRA fix versions for 3.0.0 releases]
Hi Junping, thanks for sharing your thoughts, inline, On Wed, Jul 27, 2016 at 9:10 AM, 俊平堵 wrote: > Thanks Vinod for bringing up this topic for discussion. I share the same > concern here from my previous experience and I doubt some simple rules > proposed below could make life easier. > > > The question now is what we do for the 2.8.0 and 3.0.0-alpha1 fix > versions. > > Allen's historical perspective is that we've based each minor or major > > release off of the previous minor release. So, 2.8.0 would be based off > of > > 2.7.0. Assuming 3.0.0-alpha1 happens before 2.8.0, 3.0.0-alpha1 would > also > > be based off of 2.7.0. This also makes sense from a user POV; someone on > a > > 2.6.x going to 3.0.0-alpha1 can look at the 2.7.0 and 3.0.0-alpha1 notes > to > > see what's changed. > This is not correct - not reflecting the past and not helpful for the > future. There is no benefit to claim 3.0.0-alpha1 is based on 2.7.0 over > 2.7.3 (in case 2.8.0 is not there). > In the past, for example, when we cut off 2.7, we already have 2.6.0 and > 2.6.1 get released, so 2.7.0 take all commits from 2.6.1 (not 2.6.0). In > future, assume when we start the release effor of 3.1.0 and we have 3.0.1, > 3.0.2, etc., 3.0.x should be more stable than 3.0.0-alpha, so there is > unnecessary to do everything from scratch (3.0.0-alpha). > Based on the website, 2.7.0 immediately followed 2.6.0, so that's not quite accurate. However your point does hold for 2.5.2 -> 2.6.0. Vinod already described this earlier, where for a while we only had a single chronological release line. Now though, we have the concurrent 2.6.x and 2.7.x release lines, which do fix versions independently. One goal here is a versioning scheme that extends this concurrency to additional major and minor releases, i.e. 2.8 and 3.0. It doesn't make sense to have a global ordering of releases, since not all patches belong in all branches. It also poses additional coordination cost for release management, particularly since it's hard to predict release timings. Another goal is a scheme that is easy for users to understand. I believe the above scheme accurately encodes the versioning system used by most other enterprise software. > So the rule here should be: a new major or minor release should come from > a release: > 1. tag with stable > 2. released latest > 3. with maximum version number > If condition 2 and 3 get conflicts, we should give priority to 3. For > example, when 3.0.0-alpha1 is about to release, assume we have 2.8.0, 2.7.4 > and 2.7.4 get released after 2.8.0, then we should claim 3.0.0-alpha is > based on 2.8.0 instead of 2.7.4. > > These rules seem to require a high degree of coordination when it comes to release ordering, but one goal is to reduce coordination. For example, the 2.7.0 notes say "Production users should wait for a 2.7.1/2.7.2 release." We didn't know in advance if 2.7.1 or 2.7.2 would be the stable release. We also don't know the release ordering of 2.7.4 and 2.8.0. Given the uncertainty around release timing and stability, it's advantageous to avoid basing versioning on these two pieces of information. With these rules, a changes in stability or release ordering means we need to update JIRA fix versions. I also didn't quite follow your example, since I assume that 2.8.0 will *not* be marked stable similar to how 2.7.0 is not stable. If this is true, does rule 1 take precedence and we would base 3.0.0-alpha1 off of 2.7.4 (the latest stable release)? That seems confusing given the presence of 2.8.0. Could you elaborate on what you see as the advantages of this scheme from a user POV? It seems like users now need to be aware of the total ordering and also stability of releases to know what changelogs to read. And as described previously, it's not how other enterprise software versioning works. > > As an example, if a JIRA was committed to branch-2.6, branch-2.7, > branch-2, > > branch-3.0.0-alpha1, and trunk, it could have fix versions of 2.6.5, > 2.7.3, > > 2.8.0, 3.0.0-alpha1. The first two fix versions come from application of > > rule 1, and the last two fix versions come from rule 2. > I don't think setting version tags to be more than 3 is a good practice. > The example above means we need to backport this patch to 5 branches which > make our committers' life really tough - it requires more effort of > committing a patch and also increases the risky of bugs that caused by > backport. Given realistic community review bandwidth (please check another > thread from Chris D.), I strongly suggest we keep active release train to > be less than 3, so we can have 2 stable release or 1 stable release + 1 > alpha release in releasing. > > We already need to backport to many branches, setting some more fix versions doesn't change that. Setting more fix versions is also not related to review bandwidth, particularly in this case since things normally land in trunk first. It's also not related to the number of concurrent release
Re: [VOTE] Release Apache Hadoop 2.7.3 RC0
+1 (binding) - downloaded both source and binary tarballs and verified the signatures - set up a pseudo-distributed cluster - ran some simple mapreduce jobs - checked the basic web UI Sangjin On Wed, Jul 27, 2016 at 12:57 PM, John Zhuge wrote: > +1 (non-binding) > > - Build source with Java 1.8.0_101 on Centos 7.2 with native > - Build source with Java 1.7.0_79 on Mac > - Verify license and notice using the shell script in HADOOP-13374 > - Deploy a pseudo cluster > - Run basic dfs, distcp, ACL, webhdfs commands > - Run MapReduce workcount and pi examples > - Start balancer > > Thanks, > John > > John Zhuge > Software Engineer, Cloudera > > On Wed, Jul 27, 2016 at 11:38 AM, Robert Kanter > wrote: > > > +1 (binding) > > > > - Downloaded binary tarball > > - verified signatures > > - setup pseudo cluster > > - ran some of the example jobs, clicked around the UI a bit > > > > > > - Robert > > > > > > On Mon, Jul 25, 2016 at 3:28 PM, Jason Lowe > > > wrote: > > > > > +1 (binding) > > > - Verified signatures and digests- Built from source with native > support- > > > Deployed a pseudo-distributed cluster- Ran some sample jobs > > > Jason > > > > > > From: Vinod Kumar Vavilapalli > > > To: "common-...@hadoop.apache.org" ; > > > hdfs-dev@hadoop.apache.org; yarn-...@hadoop.apache.org; " > > > mapreduce-...@hadoop.apache.org" > > > Cc: Vinod Kumar Vavilapalli > > > Sent: Friday, July 22, 2016 9:15 PM > > > Subject: [VOTE] Release Apache Hadoop 2.7.3 RC0 > > > > > > Hi all, > > > > > > I've created a release candidate RC0 for Apache Hadoop 2.7.3. > > > > > > As discussed before, this is the next maintenance release to follow up > > > 2.7.2. > > > > > > The RC is available for validation at: > > > http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/ < > > > http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/> > > > > > > The RC tag in git is: release-2.7.3-RC0 > > > > > > The maven artifacts are available via repository.apache.org < > > > http://repository.apache.org/> at > > > > https://repository.apache.org/content/repositories/orgapachehadoop-1040/ > > < > > > > https://repository.apache.org/content/repositories/orgapachehadoop-1040/ > > > > > > > > > The release-notes are inside the tar-balls at location > > > hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I > > > hosted this at > > > http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/releasenotes.html < > > > http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html> > > for > > > your quick perusal. > > > > > > As you may have noted, a very long fix-cycle for the License & Notice > > > issues (HADOOP-12893) caused 2.7.3 (along with every other Hadoop > > release) > > > to slip by quite a bit. This release's related discussion thread is > > linked > > > below: [1]. > > > > > > Please try the release and vote; the vote will run for the usual 5 > days. > > > > > > Thanks, > > > Vinod > > > > > > [1]: 2.7.3 release plan: > > > > https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html > > < > > > http://markmail.org/thread/6yv2fyrs4jlepmmr> > > > > > > > > > > > >
Apply for "assign to me" permission.
Hi guys, I have opened a JIRA(https://issues.apache.org/jira/browse/HDFS-10690) and want to fix it, but i can't assign it to myself, anyone can give me a hand? Thanks a lot! Fenghua
Re: Apply for "assign to me" permission.
Done. Thanks for your interest in contributing to Hadoop. On 7/27/16, 6:47 PM, "hufh" wrote: Hi guys, I have opened a JIRA(https://issues.apache.org/jira/browse/HDFS-10690) and want to fix it, but i can't assign it to myself, anyone can give me a hand? Thanks a lot! Fenghua
[jira] [Created] (HDFS-10699) Log object instance get incorrectly in TestDFSAdmin
Yiqun Lin created HDFS-10699: Summary: Log object instance get incorrectly in TestDFSAdmin Key: HDFS-10699 URL: https://issues.apache.org/jira/browse/HDFS-10699 Project: Hadoop HDFS Issue Type: Bug Reporter: Yiqun Lin Assignee: Yiqun Lin Priority: Minor In class TestDFSAdmin, it gets a incorrect object instance. The codes: {code} public class TestDFSAdmin { private static final Log LOG = LogFactory.getLog(DFSAdmin.class); private Configuration conf = null; private MiniDFSCluster cluster; private DFSAdmin admin; ... {code} Here the class name {{DFSAdmin}} should be {{TestDFSAdmin}}. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-10700) I increase the value of the GC_OPTS on namenode. After I modified the value ,namenode start failed.
Liu Guannan created HDFS-10700: -- Summary: I increase the value of the GC_OPTS on namenode. After I modified the value ,namenode start failed. Key: HDFS-10700 URL: https://issues.apache.org/jira/browse/HDFS-10700 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 2.7.2 Environment: Linux Suse 11 SP3 Reporter: Liu Guannan I increase the value of the GC_OPTS on namenode. After I modified the value ,namenode start failed.The reasion is that Datanodes reported block status to the namenode, resulting in namenode update block status slowly. And then namenode start failed. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: [DISCUSS] Retire BKJM from trunk?
If I remember correctly, Huawei also adopted QJM component. I hope @Vinay might have discussed internally in Huawei before starting this e-mail discussion thread. I'm +1, for removing the bkjm contrib from the trunk code. Also, there are quite few open sub-tasks under HDFS-3399 umbrella jira, which was used for the BKJM implementation time. How about closing these jira by marking as "Won't Fix"? Thanks, Rakesh Intel On Thu, Jul 28, 2016 at 1:53 AM, Sijie Guo wrote: > + Rakesh and Uma > > Rakesh and Uma might have a better idea on this. I think Huawei was using > it when Rakesh and Uma worked there. > > - Sijie > > On Wed, Jul 27, 2016 at 12:06 PM, Chris Nauroth > wrote: > > > I recommend including the BookKeeper community in this discussion. I’ve > > added their user@ and dev@ lists to this thread. > > > > I do not see BKJM being used in practice. Removing it from trunk would > be > > attractive in terms of less code for Hadoop to maintain and build, but if > > we find existing users that want to keep it, I wouldn’t object. > > > > --Chris Nauroth > > > > On 7/26/16, 11:14 PM, "Vinayakumar B" wrote: > > > > Hi All, > > > >BKJM was Active and made much stable when the NameNode HA was > > implemented and there was no QJM implemented. > >Now QJM is present and is much stable which is adopted by many > > production environment. > >I wonder whether it would be a good time to retire BKJM from > trunk? > > > >Are there any users of BKJM exists? > > > > -Vinay > > > > > > >
Re: [DISCUSS] Retire BKJM from trunk?
For Huawei, Vinay/Brahma should know about their usage. I think after QJM stabilized and ready they also adopted to QJM is what I know, but they should know more than me as I left that employer while ago. If no one is using it, It is ok to remove. Regards, Uma On 7/27/16, 9:49 PM, "Rakesh Radhakrishnan" wrote: >If I remember correctly, Huawei also adopted QJM component. I hope @Vinay >might have discussed internally in Huawei before starting this e-mail >discussion thread. I'm +1, for removing the bkjm contrib from the trunk >code. > >Also, there are quite few open sub-tasks under HDFS-3399 umbrella jira, >which was used for the BKJM implementation time. How about closing these >jira by marking as "Won't Fix"? > >Thanks, >Rakesh >Intel > >On Thu, Jul 28, 2016 at 1:53 AM, Sijie Guo wrote: > >> + Rakesh and Uma >> >> Rakesh and Uma might have a better idea on this. I think Huawei was >>using >> it when Rakesh and Uma worked there. >> >> - Sijie >> >> On Wed, Jul 27, 2016 at 12:06 PM, Chris Nauroth >> >> wrote: >> >> > I recommend including the BookKeeper community in this discussion. >>I¹ve >> > added their user@ and dev@ lists to this thread. >> > >> > I do not see BKJM being used in practice. Removing it from trunk >>would >> be >> > attractive in terms of less code for Hadoop to maintain and build, >>but if >> > we find existing users that want to keep it, I wouldn¹t object. >> > >> > --Chris Nauroth >> > >> > On 7/26/16, 11:14 PM, "Vinayakumar B" >>wrote: >> > >> > Hi All, >> > >> >BKJM was Active and made much stable when the NameNode HA was >> > implemented and there was no QJM implemented. >> >Now QJM is present and is much stable which is adopted by many >> > production environment. >> >I wonder whether it would be a good time to retire BKJM from >> trunk? >> > >> >Are there any users of BKJM exists? >> > >> > -Vinay >> > >> > >> > >> - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org