Just to gauge what amount of stuff is in branch-0.20-security-203 I wrote a quick script which does a comparison based on JIRAs mention in the commit log. It output the following list of JIRAs that are in the branch but not committed to trunk. I've marked many as N/A meaning that they don't apply to trunk:
< HADOOP-1026 < HADOOP-6304 N/A < HADOOP-6544 < HADOOP-6598 N/A < HADOOP-6638 < HADOOP-6653 N/A < HADOOP-6716 N/A < HADOOP-6718 N/A < HADOOP-6728 < HADOOP-6745 < HADOOP-6757 < HADOOP-6776 N/A < HADOOP-6808 < HADOOP-6810 N/A < HADOOP-6832 < HADOOP-6855 N/A < HADOOP-7143 N/A < HADOOP-7190 N/A < HADOOP-7232 N/A < HADOOP-7243 N/A < HADOOP-7246 (not sure about this) < HADOOP-7247 < HADOOP-7253 < HDFS-1020 N/A < HDFS-1022 < HDFS-1136 N/A < HDFS-1153 < HDFS-1158 < HDFS-1313 N/A < HDFS-495 N/A < HDFS-6599 <-- must be a typo in commit log < HDFS-740 N/A < MAPREDUCE-1088 < MAPREDUCE-1100 < MAPREDUCE-1118 < MAPREDUCE-1207 < MAPREDUCE-1361 N/A < MAPREDUCE-1376 N/A < MAPREDUCE-1442 N/A < MAPREDUCE-1521 < MAPREDUCE-1526 N/A < MAPREDUCE-1550 N/A < MAPREDUCE-1594 N/A < MAPREDUCE-1641 < MAPREDUCE-1671 < MAPREDUCE-1672 < MAPREDUCE-1676 < MAPREDUCE-1677 < MAPREDUCE-1682 < MAPREDUCE-1687 < MAPREDUCE-1699 Unclear < MAPREDUCE-1711 N/A < MAPREDUCE-1713 < MAPREDUCE-1716 < MAPREDUCE-1730 < MAPREDUCE-1741 < MAPREDUCE-1744 < MAPREDUCE-1758 < MAPREDUCE-1759 N/A < MAPREDUCE-1778 N/A < MAPREDUCE-1790 < MAPREDUCE-1807 N/A < MAPREDUCE-1854 < MAPREDUCE-1871 < MAPREDUCE-1872 < MAPREDUCE-1882 < MAPREDUCE-1889 < MAPREDUCE-1890 < MAPREDUCE-1914 < MAPREDUCE-1921 < MAPREDUCE-1933 < MAPREDUCE-1934 < MAPREDUCE-1938 < MAPREDUCE-1943 < MAPREDUCE-1954 < MAPREDUCE-1955 < MAPREDUCE-1960 < MAPREDUCE-1964 < MAPREDUCE-1966 < MAPREDUCE-1971 < MAPREDUCE-2005 N/A < MAPREDUCE-2019 < MAPREDUCE-2055 < MAPREDUCE-2316 < MAPREDUCE-2355 < MAPREDUCE-291 < MAPREDUCE-323 < MAPREDUCE-339 < MAPREDUCE-517 < MAPREDUCE-6419 <-- doesnt exist, probably typo Certainly some of these are new test cases, benchmark improvements, or system tests. But many others are large new features (e.g new metrics framework, separate JobHistory service). Others also introduce new configurations (eg new JT based limits). In the list above there are 58 that seem to be applicable, probably at least half of which are non-test code. This list above doesn't include 192 patches that were committed to the branch without reference to any JIRA in the commit message: todd@todd-w510:~/git/hadoop-common$ for x in $(git rev-list origin/branch-0.20..origin/branch-0.20-security-203 -- src) ; do git log -n1 $x | egrep -q '(MAPREDUCE|HDFS|HADOOP)[-:][0-9]+' || echo $x ; done | wc -l 192 Browsing through these, many have already been forward ported, or at least had corresponding JIRAs opened. But it's very difficult to match them up and evaluate which ones have been committed. Eli pointed out one earlier this week that was done by a non-committer with no public review that introduced an apparent performance regression; it's difficult to know whether there might be others as well. Rather than being a "maintenance release" (as is usually expected when incrementing the third component of a version string) this is essentially a separate trunk off of 0.20. I agree that the advancements in this branch are many, and are a great set of contributions for the community. User limits and security are two such that have been cited in this thread; unfortunately the new improvements in limits haven't been committed to trunk, and the security in trunk has a known root exploit. Do users really want to see us putting these things in 20 without making sure they'll also show up in future releases? Looking at recent history of 204 it seems some more patches have gone in there before going into trunk as well - for example MR-2429. Arun and Sid are working on forward-porting it, and it's obviously not due to any kind of bad intent that it was missed, but it underscores the dangers of having essentially two trunks in ASF. I completely agree that there should be long-term maintenance branches at the ASF, but we need to establish a clear process to make sure that "maintenance" doesn't diverge into something else. Here are two requests that others have made but I haven't seen an answer to yet: - Document the criteria by which developers can judge whether an improvement should be included in branch-0.20-security. The inclusion criteria for the branch as it stands is not clear -- given this branch's lineage, it clearly used to be "things important for the Yahoo clusters", but that doesn't seem like a reasonable community criterion. Up until now in Hadoop's history, the criteria has always been "compatible bug fixes only", which doesn't describe this branch either. - Clearly establish the process that all patches must either be committed to trunk first (and then backported), or include a comment on the JIRA explaining why this is not necessary. Additionally we should decide whether patches must be backported "through" 0.22 or if they may skip back from trunk to 20-security. (I'm assuming 21 is dead here) Perhaps this could go on a wiki page (or web site page) regarding the currently active branches? Thanks -Todd On Tue, May 3, 2011 at 10:02 AM, Eli Collins <e...@cloudera.com> wrote: > I think we still need to incorporate the patches currently checked > into branch 0.20. For example, Owen identified a major bug > (BooleanWritable's comparator is broken) and filed a jira > (HADOOP-6928) to put it in branch-0.20, where I reviewed it and > checked it in, so this bug would be fixed in the next stable release. > However this change is not in branch-0.20-security-203. Unless we put > the delta from branch-0.20 into this release, it is missing important > bug fixes that will cause it to regress against 20.3 (if it ever is > released). > > I am also nervous about changes like the one identified by > HADOOP-7255. It looks like this change caused a significant regression > in TestDFSIO throughput. It changes the core Task class, the commit > log is a single line, and as far as I can tell it was not discussed or > reviewed by anyone in the community. Don't changes like this at least > deserve a jira before we release them? > > Thanks, > Eli > > On Tue, May 3, 2011 at 1:39 AM, Konstantin Shvachko > <shv.had...@gmail.com> wrote: > > I think its a good idea to release hadoop-0.20.203. It moves Apache > Hadoop a > > step forward. > > > > Looks like the technical difficulties are resolved now with latest Arun's > > commits. > > Being a superset of hadoop-0.20.2 it can be considered based on one of > the > > official Apache releases. > > I don't think there was a lack of discussions on the lists about the > issues > > included in the release candidate. Todd did a thorough review of the > entire > > security branch. Many developers participated in discussions. > > Agreeing with Stack I wish HBase was considered a primary target for > Hadoop > > support. But it is not realistic to have it in hadoop-0.20.203. > > I have some experience running a version of this release candidate on a > > large cluster. It works. I would add a couple of patches, which make it > run > > on Windows for me like HADOOP-7110, HADOOP-7126. But those are not > blockers. > > > > Thanks, > > --Konstantin > > > > > > On Mon, May 2, 2011 at 5:12 PM, Ian Holsman <had...@holsman.net> wrote: > > > >> > >> On May 3, 2011, at 9:58 AM, Arun C Murthy wrote: > >> > >> >> > >> >> Owen, Suresh and I have committed everything on this list except > >> >> HADOOP-6386 and HADOOP-6428. Not sure which of the two are relevant/ > >> >> necessary, I'll check with Cos. Other than that hadoop-0.20.203 now > a > >> >> superset of hadoop-0.20.2. > >> >> > >> > > >> > Missed adding HADOOP-5759 to that list, I'll check with Amareshwari > >> before committing. > >> > > >> > Arun > >> > >> Thanks for doing this so fast Arun. > >> > >> > > > -- Todd Lipcon Software Engineer, Cloudera
