Re: [EVENT] HDFS Bug Bash: March 12
+1 Thanks for making this happens! Tsz-Wo On Tuesday, March 6, 2018, 7:49:06 PM PST, Chris Douglas wrote: Found a meetup alternative (thanks Subru): https://meetingstar.io/event/fk13172f1d75KN So we can get a rough headcount, please add (local) if you plan to attend in-person. -C On Mon, Mar 5, 2018 at 4:03 PM, Chris Douglas wrote: > [Cross-posting, as this affects the rest of the project] > > Hey folks- > > As discussed last month [1], the HDFS build hasn't been healthy > recently. We're dedicating a bug bash to stabilize the build and > address some longstanding issues with our unit tests. We rely on our > CI infrastructure to keep the project releasable, and in its current > state, it's not protecting us from regressions. While we probably > won't achieve all our goals in this session, we can develop the > conditions for reestablishing a firm foundation. > > If you're new to the project, please consider attending and > contributing. Committers often prioritize large or complicated > patches, and the issues that make the project livable don't get enough > attention. A bug bash is a great opportunity to pull reviewers' > collars, and fix the annoyances that slow us all down. > > If you're a committer, please join us! While some of the proposed > repairs are rote, many unit tests rely on implementation details and > non-obvious invariants. We need domain experts to help untangle > complex dependencies and to prevent breakage of deliberate, but > counter-intuitive code. > > We're collecting tasks in wiki [2] and will include a dial-in option > for folks who aren't local. > > Meetup has started charging for creating new events, so we'll have to > find another way to get an approximate headcount and publish the > address. Please ping me if you have a preferred alternative. -C > > [1]: https://s.apache.org/nEoQ > [2]: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=75965105 - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: IMPORTANT: automatic changelog creation
Generating change log from JIRA is a good idea. It bases on an assumption that each JIRA has an accurate summary (a.k.a. JIRA title) to reflect the committed change. Unfortunately, the assumption is invalid for many cases since we never enforce that the JIRA summary must be the same as the change log. We may compare the current CHANGES.txt with the generated change log. I beg the diff is long. Besides, after a release R1 is out, someone may (accidentally or intentionally) modify the JIRA summary. Then, the entry for the same item in a later release R2 could be different from the one in R1. I agree that manually editing CHANGES.txt is not a perfect solution. However, it works well in the past for many releases. I suggest we keep the current dev workflow. Try using the new script provided by HADOOP-11731 to generate the next release. If everything works well, we shell remove CHANGES.txt and revise the dev workflow. What do you think? Regards,Tsz-Wo On Thursday, April 2, 2015 12:57 PM, Allen Wittenauer wrote: On Apr 2, 2015, at 12:40 PM, Vinod Kumar Vavilapalli wrote: > > We'd then doing two commits for every patch. Let's simply not remove > CHANGES.txt from trunk, keep the existing dev workflow, but doc the release > process to remove CHANGES.txt in trunk at the time of a release going out of > trunk. Might as well copy branch-2’s changes.txt into trunk then. (or 2.7’s. Last I looked, people updated branch-2 and not 2.7’s or vice versa for some patches that went into both branches.) So that folks who are committing to both branches and want to cherry pick all changes can. I mean, trunk’s is very very very wrong. Right now. Today. Borderline useless. See HADOOP-11718 (which I will now close out as won’t fix)… and that jira is only what is miscategorized, not what is missing.
Re: 2.7.3 release plan
> ... I volunteer to help on releasing 2.8.0 as 2.7.3 + HDFS-8578 and HDFS-8791. HDFS-8578 (parallel upgrade), which does not change layout, should remain in 2.7.3. Only HDFS-8791 (new layout) is needed to be put in 2.8.0. Tsz-Wo On Monday, April 4, 2016 2:51 AM, Haohui Mai wrote: +1 on option 3. I volunteer to help on releasing 2.8.0 as 2.7.3 + HDFS-8578 and HDFS-8791. ~Haohui On Fri, Apr 1, 2016 at 2:54 PM, Chris Trezzo wrote: > A few thoughts: > > 1. To echo Andrew Wang, HDFS-8578 (parallel upgrades) should be a > prerequisite for HDFS-8791. Without that patch, upgrades can be very slow > for data nodes depending on your setup. > > 2. We have already deployed this patch internally so, with my Twitter hat > on, I would be perfectly happy as long as it makes it into trunk and 2.8. > That being said, I would be hesitant to deploy the current 2.7.x or 2.6.x > releases on a large production cluster that has a diverse set of block ids > without this patch, especially if your data nodes have a large number of > disks or you are using federation. To be clear though: this highly depends > on your setup and at a minimum you should verify that this regression will > not affect you. The current block-id based layout in 2.6.x and 2.7.2 has a > performance regression that gets worse over time. When you see it happening > on a live cluster, it is one of the harder issues to identify a root cause > and debug. I do understand that this is currently only affecting a smaller > number of users, but I also think this number has potential to increase as > time goes on. Maybe we can issue a warning in the release notes for future > 2.7.x and 2.6.x releases? > > 3. One option (this was suggested on HDFS-8791 and I think Sean alluded to > this proposal on this thread) would be to cut a 2.8 release off of the > 2.7.3 release with the new layout. What people currently think of as 2.8 > would then become 2.9. This would give customers a stable release that they > could deploy with the new layout and would not break upgrade and downgrade > expectations. > > On Fri, Apr 1, 2016 at 11:32 AM, Andrew Purtell wrote: > >> As a downstream consumer of Apache Hadoop 2.7.x releases, I expect we would >> patch the release to revert HDFS-8791 before pushing it out to production. >> For what it's worth. >> >> >> On Fri, Apr 1, 2016 at 11:23 AM, Andrew Wang >> wrote: >> >> > One other thing I wanted to bring up regarding HDFS-8791, we haven't >> > backported the parallel DN upgrade improvement (HDFS-8578) to branch-2.6. >> > HDFS-8578 is a very important related fix since otherwise upgrade will be >> > very slow. >> > >> > On Thu, Mar 31, 2016 at 10:35 AM, Andrew Wang >> > wrote: >> > >> > > As I expressed on HDFS-8791, I do not want to include this JIRA in a >> > > maintenance release. I've only seen it crop up on a handful of our >> > > customer's clusters, and large users like Twitter and Yahoo that seem >> to >> > be >> > > more affected are also the most able to patch this change in >> themselves. >> > > >> > > Layout upgrades are quite disruptive, and I don't think it's worth >> > > breaking upgrade and downgrade expectations when it doesn't affect the >> > (in >> > > my experience) vast majority of users. >> > > >> > > Vinod seemed to have a similar opinion in his comment on HDFS-8791, but >> > > will let him elaborate. >> > > >> > > Best, >> > > Andrew >> > > >> > > On Thu, Mar 31, 2016 at 9:11 AM, Sean Busbey >> > wrote: >> > > >> > >> As of 2 days ago, there were already 135 jiras associated with 2.7.3, >> > >> if *any* of them end up introducing a regression the inclusion of >> > >> HDFS-8791 means that folks will have cluster downtime in order to back >> > >> things out. If that happens to any substantial number of downstream >> > >> folks, or any particularly vocal downstream folks, then it is very >> > >> likely we'll lose the remaining trust of operators for rolling out >> > >> maintenance releases. That's a pretty steep cost. >> > >> >> > >> Please do not include HDFS-8791 in any 2.6.z release. Folks having to >> > >> be aware that an upgrade from e.g. 2.6.5 to 2.7.2 will fail is an >> > >> unreasonable burden. >> > >> >> > >> I agree that this fix is important, I just think we should either cut >> > >> a version of 2.8 that includes it or find a way to do it that gives an >> > >> operational path for rolling downgrade. >> > >> >> > >> On Thu, Mar 31, 2016 at 10:10 AM, Junping Du >> > wrote: >> > >> > Thanks for bringing up this topic, Sean. >> > >> > When I released our latest Hadoop release 2.6.4, the patch of >> > HDFS-8791 >> > >> haven't been committed in so that's why we didn't discuss this >> earlier. >> > >> > I remember in JIRA discussion, we treated this layout change as a >> > >> Blocker bug that fixing a significant performance regression before >> but >> > not >> > >> a normal performance improvement. And I believe HDFS community already >> > did >> > >> their best with careful and patient to
Re: Hadoop-Mapreduce-0.23-Commit - Build # 638 - Still Failing
The version is still 0.23.2 in some places, e.g. hadoop-mapreduce-project/build.xml, so that compilation failed. They should be updated to 0.23.3. Nicholas From: Apache Jenkins Server To: mapreduce-dev@hadoop.apache.org Sent: Monday, March 5, 2012 10:47 AM Subject: Hadoop-Mapreduce-0.23-Commit - Build # 638 - Still Failing See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/638/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 14895 lines...] [exec] libtool: install: /usr/bin/install -c wordcount-nopipe /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-0.23-Commit/trunk/hadoop-mapreduce-project/build/c++-examples/Linux-i386-32/bin/wordcount-nopipe [exec] libtool: install: /usr/bin/install -c pipes-sort /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-0.23-Commit/trunk/hadoop-mapreduce-project/build/c++-examples/Linux-i386-32/bin/pipes-sort [exec] make[1]: Nothing to be done for `install-data-am'. [exec] make[1]: Leaving directory `/home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-0.23-Commit/trunk/hadoop-mapreduce-project/build/c++-build/Linux-i386-32/examples/pipes' compile-c++-examples: compile-examples: ivy-resolve-test: [ivy:resolve] downloading /home/jenkins/.m2/repository/org/apache/hadoop/hadoop-hdfs/0.23.2-SNAPSHOT/hadoop-hdfs-0.23.2-SNAPSHOT-tests.jar ... [ivy:resolve] ... (1110kB) [ivy:resolve] [SUCCESSFUL ] org.apache.hadoop#hadoop-hdfs;0.23.2-SNAPSHOT!hadoop-hdfs.jar(tests) (4ms) [ivy:resolve] downloading /home/jenkins/.m2/repository/junit/junit/4.8.1/junit-4.8.1.jar ... [ivy:resolve] . (231kB) [ivy:resolve] .. (0kB) [ivy:resolve] [SUCCESSFUL ] junit#junit;4.8.1!junit.jar (3ms) [ivy:resolve] downloading http://repo1.maven.org/maven2/org/mockito/mockito-all/1.8.2/mockito-all-1.8.2.jar ... [ivy:resolve] .. ... (1315kB) [ivy:resolve] .. (0kB) [ivy:resolve] [SUCCESSFUL ] org.mockito#mockito-all;1.8.2!mockito-all.jar (1702ms) [ivy:resolve] downloading http://repo1.maven.org/maven2/org/mortbay/jetty/jetty-servlet-tester/6.1.14/jetty-servlet-tester-6.1.14.jar ... [ivy:resolve] (8kB) [ivy:resolve] .. (0kB) [ivy:resolve] [SUCCESSFUL ] org.mortbay.jetty#jetty-servlet-tester;6.1.14!jetty-servlet-tester.jar (995ms) [ivy:resolve] downloading /home/jenkins/.m2/repository/log4j/log4j/1.2.15/log4j-1.2.15.jar ... [ivy:resolve] ... (382kB)
Re: svn commit: r1304067 - in /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project: ./ bin/ conf/ hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/ hadoop-mapreduce-exam
Hi Eli, For merging a HDFS issue, please don't merge COMMON and MAPREDUCE. It generates useless merge info and emails. Does it make sense? Regards, Nicholas From: "e...@apache.org" To: mapreduce-comm...@hadoop.apache.org Sent: Thursday, March 22, 2012 2:13 PM Subject: svn commit: r1304067 - in /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project: ./ bin/ conf/ hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/ hadoop-mapreduce-examples/ hadoop-yarn/hadoop-yarn-site/ hadoop-yarn/hadoop-y... Author: eli Date: Thu Mar 22 21:13:18 2012 New Revision: 1304067 URL: http://svn.apache.org/viewvc?rev=1304067&view=rev Log: HDFS-3044. svn merge -c 1304063 from trunk Modified: hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/ (props changed) hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt (props changed) hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/bin/ (props changed) hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/conf/ (props changed) hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml (props changed) hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-examples/ (props changed) hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/ (props changed) hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/ (props changed) hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/c++/ (props changed) hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/ (props changed) hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/block_forensics/ (props changed) hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build-contrib.xml (props changed) hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build.xml (props changed) hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/data_join/ (props changed) hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/eclipse-plugin/ (props changed) hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/index/ (props changed) hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/vaidya/ (props changed) hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/examples/ (props changed) hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/java/ (props changed) hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/ (props changed) hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/fs/ (props changed) hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/hdfs/ (props changed) hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/ipc/ (props changed) hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job/ (props changed) Propchange: hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/ -- Merged /hadoop/common/trunk/hadoop-mapreduce-project:r1304063 Propchange: hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt -- Merged /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt:r1304063 Propchange: hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/bin/ -- Merged /hadoop/common/trunk/hadoop-mapreduce-project/bin:r1304063 Propchange: hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/conf/ -- Merged /hadoop/common/trunk/hadoop-mapreduce-project/conf:r1304063 Propchange: hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml -- Merged /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml:r1304063 Propchange: hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-examples/ -- Merged /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-examples:r1304063 Propchange: hadoop/common/branches/branch-0.23/hadoop-mapreduce-projec
Re: svn commit: r1304067 - in /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project: ./ bin/ conf/ hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/ hadoop-mapreduce-exam
Hi Todd, The jira number, svn revision and merge source location should be in the merge-commit message. It is very easy to find the original commit message or other details. If you think that there is a need to include the commit message, I don't mind doing it if everyone agree. One potential problem is that the merge-commit message may be too long. Beside this related subject, do you think that my previous comment make sense? Regards, Nicholas - Original Message - From: Todd Lipcon To: mapreduce-dev@hadoop.apache.org; Tsz Wo Sze Cc: hdfs dev Sent: Thursday, March 22, 2012 3:49 PM Subject: Re: svn commit: r1304067 - in /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project: ./ bin/ conf/ hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/ hadoop-mapreduce-examples/ hadoop-yarn/hadoop-yarn-site/ hadoop-yarn/ha On a related subject, can I make a request that, when we merge issues, we also include the original commit message? ie instead of just: "Merging HDFS-12345 from trunk", we say something like "Merge HDFS-12345 from trunk: Fix the blah bug." (or just re-use the original commit)? I find it really difficult to follow the log on branch-23 when many of the commits just reference JIRA numbers with no further explanation. -Todd On Thu, Mar 22, 2012 at 3:46 PM, Tsz Wo Sze wrote: > Hi Eli, > > For merging a HDFS issue, please don't merge COMMON and MAPREDUCE. It > generates useless merge info and emails. Does it make sense? > > Regards, > > Nicholas > > > > > From: "e...@apache.org" > To: mapreduce-comm...@hadoop.apache.org > Sent: Thursday, March 22, 2012 2:13 PM > Subject: svn commit: r1304067 - in > /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project: ./ bin/ conf/ > hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/ > hadoop-mapreduce-examples/ hadoop-yarn/hadoop-yarn-site/ > hadoop-yarn/hadoop-y... > > Author: eli > Date: Thu Mar 22 21:13:18 2012 > New Revision: 1304067 > > URL: http://svn.apache.org/viewvc?rev=1304067&view=rev > Log: > HDFS-3044. svn merge -c 1304063 from trunk > > Modified: > hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/ (props > changed) > hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt > (props changed) > hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/bin/ (props > changed) > hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/conf/ > (props changed) > > hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml > (props changed) > > hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-examples/ > (props changed) > > hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/ > (props changed) > > hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/ > (props changed) > hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/c++/ > (props changed) > hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/ > (props changed) > > hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/block_forensics/ > (props changed) > > hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build-contrib.xml > (props changed) > > hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build.xml > (props changed) > > hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/data_join/ > (props changed) > > hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/eclipse-plugin/ > (props changed) > > hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/index/ > (props changed) > > hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/vaidya/ > (props changed) > > hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/examples/ > (props changed) > hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/java/ > (props changed) > > hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/ > (props changed) > > hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/fs/ > (props changed) > > hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/hdfs/ > (props changed)
Re: svn commit: r1304067 - in /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project: ./ bin/ conf/ hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/ hadoop-mapreduce-exam
Hi Eli, Checking in a script to me seems unnecessary since the svn command is already simple. It just likes that we don't check in a script for committing patches. We already have a wiki (http://wiki.apache.org/hadoop/HowToCommit) for committing and merging. I am not sure if you are replying my email. Does my previous comment make sense to you? Regards, Nicholas - Original Message - From: Eli Collins To: hdfs-...@hadoop.apache.org Cc: mapreduce-dev@hadoop.apache.org; Tsz Wo Sze Sent: Thursday, March 22, 2012 4:12 PM Subject: Re: svn commit: r1304067 - in /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project: ./ bin/ conf/ hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/ hadoop-mapreduce-examples/ hadoop-yarn/hadoop-yarn-site/ hadoop-yarn/ha Works for me. How about we check a merge script into the repo so we all do merges the same way? On Thu, Mar 22, 2012 at 3:49 PM, Todd Lipcon wrote: > On a related subject, can I make a request that, when we merge issues, > we also include the original commit message? > ie instead of just: "Merging HDFS-12345 from trunk", we say something > like "Merge HDFS-12345 from trunk: Fix the blah bug." (or just re-use > the original commit)? > > I find it really difficult to follow the log on branch-23 when many of > the commits just reference JIRA numbers with no further explanation. > > -Todd > > On Thu, Mar 22, 2012 at 3:46 PM, Tsz Wo Sze wrote: >> Hi Eli, >> >> For merging a HDFS issue, please don't merge COMMON and MAPREDUCE. It >> generates useless merge info and emails. Does it make sense? >> >> Regards, >> >> Nicholas >> >> >> >> >> From: "e...@apache.org" >> To: mapreduce-comm...@hadoop.apache.org >> Sent: Thursday, March 22, 2012 2:13 PM >> Subject: svn commit: r1304067 - in >> /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project: ./ bin/ conf/ >> hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/ >> hadoop-mapreduce-examples/ hadoop-yarn/hadoop-yarn-site/ >> hadoop-yarn/hadoop-y... >> >> Author: eli >> Date: Thu Mar 22 21:13:18 2012 >> New Revision: 1304067 >> >> URL: http://svn.apache.org/viewvc?rev=1304067&view=rev >> Log: >> HDFS-3044. svn merge -c 1304063 from trunk >> >> Modified: >> hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/ (props >> changed) >> hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt >> (props changed) >> hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/bin/ >> (props changed) >> hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/conf/ >> (props changed) >> >> hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml >> (props changed) >> >> hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-examples/ >> (props changed) >> >> hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/ >> (props changed) >> >> hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/ >> (props changed) >> hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/c++/ >> (props changed) >> >> hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/ >> (props changed) >> >> hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/block_forensics/ >> (props changed) >> >> hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build-contrib.xml >> (props changed) >> >> hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build.xml >> (props changed) >> >> hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/data_join/ >> (props changed) >> >> hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/eclipse-plugin/ >> (props changed) >> >> hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/index/ >> (props changed) >> >> hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/vaidya/ >> (props changed) >> >> hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/examples/ >> (props changed) >> hadoop/common/branches/branch-0.23/hadoop-mapreduce-pro
Re: svn commit: r1304067 - in /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project: ./ bin/ conf/ hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/ hadoop-mapreduce-exam
Hi Harsh, $svn --version svn, version 1.6.17 (r1128011) compiled Nov 8 2011, 18:14:46 I think the version is irrelevant. You have to merge under the sub-project directory, i.e. "svn merge -c /path/to/trunk/hadoop-hdfs-project/hadoop-hdfs .", provided that all the changes are that sub-directory. Regards, Nicholas - Original Message - From: Harsh J To: mapreduce-dev@hadoop.apache.org; Tsz Wo Sze Cc: hdfs dev Sent: Thursday, March 22, 2012 9:45 PM Subject: Re: svn commit: r1304067 - in /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project: ./ bin/ conf/ hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/ hadoop-mapreduce-examples/ hadoop-yarn/hadoop-yarn-site/ hadoop-yarn/ha This comes up when an svn merge command itself is carried out. The particular list of files actually is always the same. It may be that these files are set to track mergeinfo or something (Maybe someone touched them long ago with a different version of svn?). What version of svn do you use Nicholas? Do you not see this when you run the "svn merge -c /path/to/trunk ." command? On Fri, Mar 23, 2012 at 4:16 AM, Tsz Wo Sze wrote: > Hi Eli, > > For merging a HDFS issue, please don't merge COMMON and MAPREDUCE. It > generates useless merge info and emails. Does it make sense? > > Regards, > > Nicholas > > > > > From: "e...@apache.org" > To: mapreduce-comm...@hadoop.apache.org > Sent: Thursday, March 22, 2012 2:13 PM > Subject: svn commit: r1304067 - in > /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project: ./ bin/ conf/ > hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/ > hadoop-mapreduce-examples/ hadoop-yarn/hadoop-yarn-site/ > hadoop-yarn/hadoop-y... > > Author: eli > Date: Thu Mar 22 21:13:18 2012 > New Revision: 1304067 > > URL: http://svn.apache.org/viewvc?rev=1304067&view=rev > Log: > HDFS-3044. svn merge -c 1304063 from trunk > > Modified: > hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/ (props > changed) > hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt > (props changed) > hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/bin/ (props > changed) > hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/conf/ > (props changed) > > hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml > (props changed) > > hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-examples/ > (props changed) > > hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/ > (props changed) > > hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/ > (props changed) > hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/c++/ > (props changed) > hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/ > (props changed) > > hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/block_forensics/ > (props changed) > > hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build-contrib.xml > (props changed) > > hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build.xml > (props changed) > > hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/data_join/ > (props changed) > > hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/eclipse-plugin/ > (props changed) > > hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/index/ > (props changed) > > hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/vaidya/ > (props changed) > > hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/examples/ > (props changed) > hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/java/ > (props changed) > > hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/ > (props changed) > > hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/fs/ > (props changed) > > hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/hdfs/ > (props changed) > > hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/ipc/ > (props changed) > > hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps
Re: [Vote] Merge branch-trunk-win to trunk
Hi Konstantin S, I accidentally merged some windows bug fixes to branch-2. However, branch-2 did not compile -- it showed that the windows changes were missing in branch-2. The ones causing compilation problems were already reverted (Thanks Sid and Suresh.) Sorry for the inconvenience. Tsz-Wo From: Konstantin Shvachko To: mapreduce-dev@hadoop.apache.org; hdfs-...@hadoop.apache.org; common-...@hadoop.apache.org; yarn-...@hadoop.apache.org Sent: Tuesday, March 26, 2013 5:25 AM Subject: Re: [Vote] Merge branch-trunk-win to trunk Andrew, this used to be on all -dev lists. Let's keep it that way. To the point. Does this mean that people are silently porting windows changes to branch-2? New features on a branch should be voted first, no? Thanks, --Konstantin On Mon, Mar 25, 2013 at 1:36 PM, Andrew Purtell wrote: > Noticed this too. Simply a 'public' modifier is missing, but it's unclear > how this could not have been caught prior to check-in. > > > On Mon, Mar 25, 2013 at 9:17 PM, Konstantin Boudnik wrote: > >> It doesn't look like any progress has been done on the ticket below in the >> last 3 weeks. And now branch-2 can't be compiled because of >> >> >> hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java:[895,15] >> WINDOWS is not public in org.apache.hadoop.fs.Path; cannot be accessed from >> outside package >> >> That's exactly why I was -1'ing this... >> Cos >> >> On Mon, Mar 04, 2013 at 05:41PM, Matt Foley wrote: >> > Thanks, gentlemen. I've opened and taken responsibility for >> > https://issues.apache.org/jira/browse/HADOOP-9359. Giri Kesavan has >> agreed >> > to help with the parts that require Jenkins admin access. >> > >> > Thanks, >> > --Matt >> > >> > >> > >> > On Mon, Mar 4, 2013 at 5:00 PM, Konstantin Shvachko < >> shv.had...@gmail.com>wrote: >> > >> > > +1 on the merge. >> > > >> > > I am glad we agreed. >> > > Having Jira to track the CI effort is a good idea. >> > > >> > > Thanks, >> > > --Konstantin >> > > >> > > On Mon, Mar 4, 2013 at 3:29 PM, Matt Foley >> wrote: >> > > > Thanks. I agree Windows -1's in test-patch should not block commits. >> > > > >> > > > --Matt >> > > > >> > > > >> > > > >> > > > On Mon, Mar 4, 2013 at 2:30 PM, Konstantin Shvachko < >> > > shv.had...@gmail.com> >> > > > wrote: >> > > >> >> > > >> On Mon, Mar 4, 2013 at 12:22 PM, Matt Foley > > >> > > >> wrote: >> > > >> > Konstantine, you have voted -1, and stated some requirements >> before >> > > >> > you'll >> > > >> > withdraw that -1. As I plan to do work to fulfill those >> > > requirements, I >> > > >> > want to make sure that what I'm proposing will, in fact, satisfy >> you. >> > > >> > That's why I'm asking, if we implement full "test-patch" >> integration >> > > for >> > > >> > Windows, does it seem to you that that would provide adequate >> support? >> > > >> >> > > >> Yes. >> > > >> >> > > >> > I have learned not to presume that my interpretation is correct. >> My >> > > >> > interpretation of item #1 is that test-patch provides pre-commit >> > > build, >> > > >> > so >> > > >> > it would satisfy item #1. But rather than assuming that I am >> > > >> > interpreting >> > > >> > it correctly, I simply want your agreement that it would, or if >> not, >> > > >> > clarification why it won't. >> > > >> >> > > >> I agree it will satisfy my item #1. >> > > >> I did not agree in my previous email, but I changed my mind based on >> > > >> the latest discussion. I have to explain why now. >> > > >> I was proposing nightly build because I did not want pre-commit >> build >> > > >> for Windows block commits to Linux. But if people are fine just >> ignoring >> > > >> -1s for the Windows part of the build it should be good. >> > > >> >> > > >> > Regarding item #2, it is also my interpretation that test-patch >> > > provides >> > > >> > an >> > > >> > on-demand (perhaps 20-minutes deferred) Jenkins build and unit >> test, >> > > >> > with >> > > >> > logs available to the developer, so it would satisfy item #2. But >> > > >> > rather >> > > >> > than assuming that I am interpreting it correctly, I simply want >> your >> > > >> > agreement that it would, or if not, clarification why it won't. >> > > >> >> > > >> It will satisfy my item #2 in the following way: >> > > >> I can duplicate your pre-commit build for Windows and add an input >> > > >> parameter, which would let people run the build on their patches >> > > >> chosen from local machine rather than attaching them to Jiras. >> > > >> >> > > >> Thanks, >> > > >> --Konstantin >> > > >> >> > > >> > In agile terms, you are the Owner of these requirements. Please >> give >> > > me >> > > >> > owner feedback as to whether my proposed work sounds like it will >> > > >> > satisfy >> > > >> > the requirements. >> > > >> > >> > > >> > Thank you, >> > > >> > --Matt >> > > >> > >> > > >> > >> > > >> > On Sun, Mar 3, 2013 at 12:16 PM, Konstantin Shvachko >> > > >> > >> > > >> >
Re: svn commit: r1586494 - in /hadoop/common/branches/branch-2/hadoop-mapreduce-project: ./ CHANGES.txt bin/ conf/ hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-defaul
Hi Andrew, For merging a change, please merge only the sub-projects involved. In particular, for merging a HDFS commit, cd hadoop-hdfs-project/hadoop-hdfs and run the merge command there. Please do not merge common/mapreduce. It generates a lot of noise; see attached message at the end. Thanks. Tsz-Wo On Thursday, April 10, 2014 3:37 PM, "w...@apache.org" wrote: Author: wang >Date: Thu Apr 10 22:36:36 2014 >New Revision: 1586494 > >URL: http://svn.apache.org/r1586494 >Log: >HDFS-6224. Add a unit test to TestAuditLogger for file permissions passed to >logAuditEvent. Contributed by Charles Lamb. > >Modified: > hadoop/common/branches/branch-2/hadoop-mapreduce-project/ (props changed) > hadoop/common/branches/branch-2/hadoop-mapreduce-project/CHANGES.txt >(props changed) > hadoop/common/branches/branch-2/hadoop-mapreduce-project/bin/ (props >changed) > hadoop/common/branches/branch-2/hadoop-mapreduce-project/conf/ (props >changed) > >hadoop/common/branches/branch-2/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml > (props changed) > >hadoop/common/branches/branch-2/hadoop-mapreduce-project/hadoop-mapreduce-examples/ > (props changed) > >Propchange: hadoop/common/branches/branch-2/hadoop-mapreduce-project/ >-- > Merged /hadoop/common/trunk/hadoop-mapreduce-project:r1586490 > >Propchange: >hadoop/common/branches/branch-2/hadoop-mapreduce-project/CHANGES.txt >-- > Merged /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt:r1586490 > >Propchange: hadoop/common/branches/branch-2/hadoop-mapreduce-project/bin/ >-- > Merged /hadoop/common/trunk/hadoop-mapreduce-project/bin:r1586490 > >Propchange: hadoop/common/branches/branch-2/hadoop-mapreduce-project/conf/ >-- > Merged /hadoop/common/trunk/hadoop-mapreduce-project/conf:r1586490 > >Propchange: >hadoop/common/branches/branch-2/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml >-- > Merged >/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml:r1586490 > >Propchange: >hadoop/common/branches/branch-2/hadoop-mapreduce-project/hadoop-mapreduce-examples/ >-- > Merged >/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-examples:r1586490 > > > > >
Re: [VOTE] Release Apache Hadoop 2.6.0
I agree that HDFS-6833 is not a blocker. It is not a problem for the blocks with replication >= 3 and it is not a regression (please correct me if I am wrong.)Tsz-Wo On Monday, November 17, 2014 10:06 AM, Suresh Srinivas wrote: Tsuyoshi, thanks for bringing up HDFS-6833. However, given it is a boundary condition (and should not cause issues when for files with replication factor 3), we should perhaps target this into 2.6.1 and not block this release. Thoughts? On Mon, Nov 17, 2014 at 4:17 AM, Tsuyoshi OZAWA wrote: > +0(non-binding) > > HDFS-6833 is critical issue for us - could you help us to merge it into > 2.6? > > Thanks, > - Tsuyoshi > > On Sat, Nov 15, 2014 at 1:03 PM, Hitesh Shah wrote: > > +1 (binding) > > > > Built Hadoop from source, compiled Tez against the hadoop jars pushed to > staging repo and ran a few example Tez jobs on a single node cluster. > > > > — HItesh > > > > > > On Nov 13, 2014, at 3:08 PM, Arun C Murthy wrote: > > > >> Folks, > >> > >> I've created another release candidate (rc1) for hadoop-2.6.0 based on > the feedback. > >> > >> The RC is available at: > http://people.apache.org/~acmurthy/hadoop-2.6.0-rc1 > >> The RC tag in git is: release-2.6.0-rc1 > >> > >> The maven artifacts are available via repository.apache.org at > https://repository.apache.org/content/repositories/orgapachehadoop-1013. > >> > >> Please try the release and vote; the vote will run for the usual 5 days. > >> > >> thanks, > >> Arun > >> > >> > >> -- > >> CONFIDENTIALITY NOTICE > >> NOTICE: This message is intended for the use of the individual or > entity to > >> which it is addressed and may contain information that is confidential, > >> privileged and exempt from disclosure under applicable law. If the > reader > >> of this message is not the intended recipient, you are hereby notified > that > >> any printing, copying, dissemination, distribution, disclosure or > >> forwarding of this communication is strictly prohibited. If you have > >> received this communication in error, please contact the sender > immediately > >> and delete it from your system. Thank You. > > > > > > -- > - Tsuyoshi > -- http://hortonworks.com/download/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: [VOTE] Release Apache Hadoop 2.6.0
Hi Arun, Only md5 is provided but not mds? Is it intentional? Tsz-Wo On Monday, November 17, 2014 2:14 PM, Tsz Wo Sze wrote: I agree that HDFS-6833 is not a blocker. It is not a problem for the blocks with replication >= 3 and it is not a regression (please correct me if I am wrong.)Tsz-Wo On Monday, November 17, 2014 10:06 AM, Suresh Srinivas wrote: Tsuyoshi, thanks for bringing up HDFS-6833. However, given it is a boundary condition (and should not cause issues when for files with replication factor 3), we should perhaps target this into 2.6.1 and not block this release. Thoughts? On Mon, Nov 17, 2014 at 4:17 AM, Tsuyoshi OZAWA wrote: > +0(non-binding) > > HDFS-6833 is critical issue for us - could you help us to merge it into > 2.6? > > Thanks, > - Tsuyoshi > > On Sat, Nov 15, 2014 at 1:03 PM, Hitesh Shah wrote: > > +1 (binding) > > > > Built Hadoop from source, compiled Tez against the hadoop jars pushed to > staging repo and ran a few example Tez jobs on a single node cluster. > > > > — HItesh > > > > > > On Nov 13, 2014, at 3:08 PM, Arun C Murthy wrote: > > > >> Folks, > >> > >> I've created another release candidate (rc1) for hadoop-2.6.0 based on > the feedback. > >> > >> The RC is available at: > http://people.apache.org/~acmurthy/hadoop-2.6.0-rc1 > >> The RC tag in git is: release-2.6.0-rc1 > >> > >> The maven artifacts are available via repository.apache.org at > https://repository.apache.org/content/repositories/orgapachehadoop-1013. > >> > >> Please try the release and vote; the vote will run for the usual 5 days. > >> > >> thanks, > >> Arun > >> > >> > >> -- > >> CONFIDENTIALITY NOTICE > >> NOTICE: This message is intended for the use of the individual or > entity to > >> which it is addressed and may contain information that is confidential, > >> privileged and exempt from disclosure under applicable law. If the > reader > >> of this message is not the intended recipient, you are hereby notified > that > >> any printing, copying, dissemination, distribution, disclosure or > >> forwarding of this communication is strictly prohibited. If you have > >> received this communication in error, please contact the sender > immediately > >> and delete it from your system. Thank You. > > > > > > -- > - Tsuyoshi > -- http://hortonworks.com/download/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: [VOTE] Release Apache Hadoop 2.6.0
+1 (suppose using md5 is intentional.) Verified md5 and ran a single node HA HDFS cluster for some time. Everything worked great. Regards,Tsz-Wo On Monday, November 17, 2014 4:48 PM, Tsz Wo Sze wrote: Hi Arun, Only md5 is provided but not mds? Is it intentional? Tsz-Wo On Monday, November 17, 2014 2:14 PM, Tsz Wo Sze wrote: I agree that HDFS-6833 is not a blocker. It is not a problem for the blocks with replication >= 3 and it is not a regression (please correct me if I am wrong.)Tsz-Wo On Monday, November 17, 2014 10:06 AM, Suresh Srinivas wrote: Tsuyoshi, thanks for bringing up HDFS-6833. However, given it is a boundary condition (and should not cause issues when for files with replication factor 3), we should perhaps target this into 2.6.1 and not block this release. Thoughts? On Mon, Nov 17, 2014 at 4:17 AM, Tsuyoshi OZAWA wrote: > +0(non-binding) > > HDFS-6833 is critical issue for us - could you help us to merge it into > 2.6? > > Thanks, > - Tsuyoshi > > On Sat, Nov 15, 2014 at 1:03 PM, Hitesh Shah wrote: > > +1 (binding) > > > > Built Hadoop from source, compiled Tez against the hadoop jars pushed to > staging repo and ran a few example Tez jobs on a single node cluster. > > > > — HItesh > > > > > > On Nov 13, 2014, at 3:08 PM, Arun C Murthy wrote: > > > >> Folks, > >> > >> I've created another release candidate (rc1) for hadoop-2.6.0 based on > the feedback. > >> > >> The RC is available at: > http://people.apache.org/~acmurthy/hadoop-2.6.0-rc1 > >> The RC tag in git is: release-2.6.0-rc1 > >> > >> The maven artifacts are available via repository.apache.org at > https://repository.apache.org/content/repositories/orgapachehadoop-1013. > >> > >> Please try the release and vote; the vote will run for the usual 5 days. > >> > >> thanks, > >> Arun > >> > >> > >> -- > >> CONFIDENTIALITY NOTICE > >> NOTICE: This message is intended for the use of the individual or > entity to > >> which it is addressed and may contain information that is confidential, > >> privileged and exempt from disclosure under applicable law. If the > reader > >> of this message is not the intended recipient, you are hereby notified > that > >> any printing, copying, dissemination, distribution, disclosure or > >> forwarding of this communication is strictly prohibited. If you have > >> received this communication in error, please contact the sender > immediately > >> and delete it from your system. Thank You. > > > > > > -- > - Tsuyoshi > -- http://hortonworks.com/download/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: Hadoop-Mapreduce-trunk-Commit - Build # 750 - Failure
I have temporarily disabled TestBlockPlacementPolicyRaid. Please see https://issues.apache.org/jira/browse/MAPREDUCE-2711 Regards, Tsz-Wo From: Apache Jenkins Server To: mapreduce-dev@hadoop.apache.org Sent: Tuesday, July 19, 2011 9:08 AM Subject: Hadoop-Mapreduce-trunk-Commit - Build # 750 - Failure See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/750/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 8868 lines...] [javac] /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/src/contrib/raid/src/test/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockPlacementPolicyRaid.java:85: blockManager is not public in org.apache.hadoop.hdfs.server.namenode.FSNamesystem; cannot be accessed from outside package [javac] policy = (BlockPlacementPolicyRaid) namesystem.blockManager.replicator; [javac] ^ [javac] /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/src/contrib/raid/src/test/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockPlacementPolicyRaid.java:272: cannot find symbol [javac] symbol : variable clusterMap [javac] location: class org.apache.hadoop.hdfs.server.namenode.FSNamesystem [javac] conf, namesystem, namesystem.clusterMap)); [javac] ^ [javac] /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/src/contrib/raid/src/test/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockPlacementPolicyRaid.java:317: blockManager is not public in org.apache.hadoop.hdfs.server.namenode.FSNamesystem; cannot be accessed from outside package [javac] namesystem.blockManager.replicator = policy; [javac] ^ [javac] /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/src/contrib/raid/src/test/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockPlacementPolicyRaid.java:317: cannot assign a value to final variable replicator [javac] namesystem.blockManager.replicator = policy; [javac] ^ [javac] /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/src/contrib/raid/src/test/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockPlacementPolicyRaid.java:337: cannot find symbol [javac] symbol : variable clusterMap [javac] location: class org.apache.hadoop.hdfs.server.namenode.FSNamesystem [javac] conf, namesystem, namesystem.clusterMap)); [javac] ^ [javac] /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/src/contrib/raid/src/test/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockPlacementPolicyRaid.java:454: cannot find symbol [javac] symbol : variable clusterMap [javac] location: class org.apache.hadoop.hdfs.server.namenode.FSNamesystem [javac] policy.initialize(conf, namesystem, namesystem.clusterMap); [javac] ^ [javac] /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/src/contrib/raid/src/test/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockPlacementPolicyRaid.java:501: blockManager is not public in org.apache.hadoop.hdfs.server.namenode.FSNamesystem; cannot be accessed from outside package [javac] INodeFile inode = namesystem.blockManager.blocksMap.getINode(block [javac] ^ [javac] Note: Some input files use or override a deprecated API. [javac] Note: Recompile with -Xlint:deprecation for details. [javac] 8 errors BUILD FAILED /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/build.xml:1189: The following error occurred while executing this line: /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/src/contrib/build.xml:39: The following error occurred while executing this line: /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/src/contrib/build-contrib.xml:227: Compile failed; see the compiler error output for details. Total time: 5 minutes 38 seconds == == STORE: saving artifacts == == mv: cannot stat `build/*.tar.gz': No such file or directory mv: cannot stat `build/test/findbugs': No such file or directory Build Failed [FINDBUGS] Skipping publisher since build result is FAILURE Archiving artifacts Publishing Clover coverage report... No Clover report will be published due to a Bui
Re: [VOTE] Merge MR-279 to trunk.
+1 Tsz-Wo From: Chris Douglas To: mapreduce-dev@hadoop.apache.org Sent: Wednesday, August 17, 2011 5:08 PM Subject: Re: [VOTE] Merge MR-279 to trunk. +1 -C On Tue, Aug 16, 2011 at 2:14 PM, Mahadev Konar wrote: > Hi all, > > We are excited to let you know that we have MR-279 ready to be merged to > trunk. I have uploaded necessary details on > https://issues.apache.org/jira/browse/MAPREDUCE-279. > > Please take a look and vote. > > Clearly I am +1 on it. > > thanks > mahadev
Re: mysterious NumberFormatException
Hi, It looks like that the ppid is a 64-bit positive integer but Java long is signed and so only works with 63-bit positive integers. In your case, 2^64 > 18446743988060683582 > 2^63. Therefore, there is a NFE. I think it is a bug in ProcfsBasedProcessTree. Regards, Nicholas Sze From: Ted Yu To: mapreduce-dev@hadoop.apache.org Cc: giridharan kesavan Sent: Monday, December 19, 2011 8:24 PM Subject: mysterious NumberFormatException Hi, HBase PreCommit builds frequently gave us mysterious NumberFormatException From https://builds.apache.org/job/PreCommit-HBASE-Build/553//testReport/org.apache.hadoop.hbase.mapreduce/TestHFileOutputFormat/testMRIncrementalLoad/ : 2011-12-20 01:44:01,180 WARN [main] mapred.JobClient(784): No job jar file set. User classes may not be found. See JobConf(Class) or JobConf#setJar(String). java.lang.NumberFormatException: For input string: "18446743988060683582" at java.lang.NumberFormatException.forInputString(NumberFormatException.java:48) at java.lang.Long.parseLong(Long.java:422) at java.lang.Long.parseLong(Long.java:468) at org.apache.hadoop.util.ProcfsBasedProcessTree.constructProcessInfo(ProcfsBasedProcessTree.java:413) at org.apache.hadoop.util.ProcfsBasedProcessTree.getProcessTree(ProcfsBasedProcessTree.java:148) at org.apache.hadoop.util.LinuxResourceCalculatorPlugin.getProcResourceValues(LinuxResourceCalculatorPlugin.java:401) at org.apache.hadoop.mapred.Task.initialize(Task.java:536) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:353) at org.apache.hadoop.mapred.Child$4.run(Child.java:255) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1083) at org.apache.hadoop.mapred.Child.main(Child.java:249) From hadoop 0.20.205 source code, looks like ppid was 18446743988060683582, causing NFE: // Set (name) (ppid) (pgrpId) (session) (utime) (stime) (vsize) (rss) pinfo.updateProcessInfo(m.group(2), Integer.parseInt(m.group(3)), You can find information on the OS at the beginning of https://builds.apache.org/job/PreCommit-HBASE-Build/553/console: asf011.sp2.ygridcore.net Linux asf011.sp2.ygridcore.net 2.6.32-33-server #71-Ubuntu SMP Wed Jul 20 17:42:25 UTC 2011 x86_64 GNU/Linux core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 20 file size (blocks, -f) unlimited pending signals (-i) 16382 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 6 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 2048 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited 6 Running in Jenkins mode Your insight is welcome.
Re: mysterious NumberFormatException
Hi Ted, Use String is a good idea. Or we may use Long but we have to be more careful to parse 64-bit integers (use negative long when >= 2^63.) Either way is fine. Could you file a JIRA? Thanks for catching the bug. Nicholas From: Ted Yu To: mapreduce-dev@hadoop.apache.org; Tsz Wo Sze Cc: giridharan kesavan Sent: Monday, December 19, 2011 11:20 PM Subject: Re: mysterious NumberFormatException Thanks for the analysis, Nicolas. Is it reasonable to change allProcessInfo to Map so that we don't encounter this problem by avoiding parsing large integer ? On Mon, Dec 19, 2011 at 9:59 PM, Tsz Wo Sze wrote: > Hi, > > It looks like that the ppid is a 64-bit positive integer but Java long is > signed and so only works with 63-bit positive integers. In your case, > > 2^64 > 18446743988060683582 > 2^63. > > Therefore, there is a NFE. I think it is a bug in ProcfsBasedProcessTree. > > > Regards, > > Nicholas Sze > > > > > From: Ted Yu > To: mapreduce-dev@hadoop.apache.org > Cc: giridharan kesavan > Sent: Monday, December 19, 2011 8:24 PM > Subject: mysterious NumberFormatException > > Hi, > HBase PreCommit builds frequently gave us mysterious NumberFormatException > > From > > https://builds.apache.org/job/PreCommit-HBASE-Build/553//testReport/org.apache.hadoop.hbase.mapreduce/TestHFileOutputFormat/testMRIncrementalLoad/ > : > > 2011-12-20 01:44:01,180 WARN [main] mapred.JobClient(784): No job jar > file set. User classes may not be found. See JobConf(Class) or > JobConf#setJar(String). > java.lang.NumberFormatException: For input string: "18446743988060683582" > at > java.lang.NumberFormatException.forInputString(NumberFormatException.java:48) > at java.lang.Long.parseLong(Long.java:422) > at java.lang.Long.parseLong(Long.java:468) > at > org.apache.hadoop.util.ProcfsBasedProcessTree.constructProcessInfo(ProcfsBasedProcessTree.java:413) > at > org.apache.hadoop.util.ProcfsBasedProcessTree.getProcessTree(ProcfsBasedProcessTree.java:148) > at > org.apache.hadoop.util.LinuxResourceCalculatorPlugin.getProcResourceValues(LinuxResourceCalculatorPlugin.java:401) > at org.apache.hadoop.mapred.Task.initialize(Task.java:536) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:353) > at org.apache.hadoop.mapred.Child$4.run(Child.java:255) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1083) > at org.apache.hadoop.mapred.Child.main(Child.java:249) > > From hadoop 0.20.205 source code, looks like ppid was 18446743988060683582, > causing NFE: > // Set (name) (ppid) (pgrpId) (session) (utime) (stime) (vsize) > (rss) > pinfo.updateProcessInfo(m.group(2), Integer.parseInt(m.group(3)), > > You can find information on the OS at the beginning of > https://builds.apache.org/job/PreCommit-HBASE-Build/553/console: > > asf011.sp2.ygridcore.net > Linux asf011.sp2.ygridcore.net 2.6.32-33-server #71-Ubuntu SMP Wed Jul > 20 17:42:25 UTC 2011 x86_64 GNU/Linux > core file size (blocks, -c) 0 > data seg size (kbytes, -d) unlimited > scheduling priority (-e) 20 > file size (blocks, -f) unlimited > pending signals (-i) 16382 > max locked memory (kbytes, -l) 64 > max memory size (kbytes, -m) unlimited > open files (-n) 6 > pipe size (512 bytes, -p) 8 > POSIX message queues (bytes, -q) 819200 > real-time priority (-r) 0 > stack size (kbytes, -s) 8192 > cpu time (seconds, -t) unlimited > max user processes (-u) 2048 > virtual memory (kbytes, -v) unlimited > file locks (-x) unlimited > 6 > Running in Jenkins mode > > Your insight is welcome. >
