Re: Code guidelines and bash
On Sun, Jul 27, 2014 at 9:28 PM, Ted Dunning tdunn...@maprtech.com wrote: I don't know of any dev environments in common use today that can't display 100 characters. I edit in an 80-column Emacs window that just fits beside an 80-column shell window on a portrait-rotated 24 monitor. Doug
Re: Change proposal for FileInputFormat isSplitable
On Fri, May 30, 2014 at 11:05 PM, Niels Basjes ni...@basjes.nl wrote: How would someone create the situation you are referring to? By renaming files. I believe I can easily write a test using public APIs that will succeed without this patch and will fail with this patch. Given the number of Hadoop-based applications now in the world, the likelihood that some perform such file renamings is non-negligible. Doug
Re: Change proposal for FileInputFormat isSplitable
I was trying to explain my comment, where I stated that, changing the default implementation to return false would be an incompatible change. The patch was added 6 months after that comment, so the comment didn't address the patch. The patch does not appear to change the default implementation to return false unless the suffix of the file name is that of a known unsplittable compression format. So the folks who'd be harmed by this are those who used a suffix like .gz for an Avro, Parquet or other-format file. Their applications might suddenly run much slower and it would be difficult for them to determine why. Such folks are probably few, but perhaps exist. I'd prefer a change that avoided that possibility entirely. Doug On Fri, May 30, 2014 at 3:02 PM, Niels Basjes ni...@basjes.nl wrote: Hi, The way I see the effects of the original patch on existing subclasses: - implemented isSplitable -- no performance difference. - did not implement isSplitable -- then there is no performance difference if the container is either not compressed or uses a splittable compression. -- If it uses a common non splittable compression (like gzip) then the output will suddenly be different (which is the correct answer) and the jobs will finish sooner because the input is not processed multiple times. Where do you see a performance impact? Niels On May 30, 2014 8:06 PM, Doug Cutting cutt...@apache.org wrote: On Thu, May 29, 2014 at 2:47 AM, Niels Basjes ni...@basjes.nl wrote: For arguments I still do not fully understand this was rejected by Todd and Doug. Performance is a part of compatibility. Doug
Re: Re-swizzle 2.3
On Wed, Jan 29, 2014 at 12:30 PM, Jason Lowe jl...@yahoo-inc.com wrote: It is a bit concerning that the JIRA history showed that the target version was set at some point in the past but no record of it being cleared. Perhaps the version itself was renamed? Doug
Re: [DISCUSS] What is the purpose of merge vote threads?
On Fri, Oct 25, 2013 at 9:35 AM, Suresh Srinivas sur...@hortonworks.com wrote: Granted some of the feature readiness activity can be done during voting. But I fail to understand why expediting a feature that takes months to build should try to optimize a week. Why not finish the requirements we have for every patch for feature branch also? I agree that requirements should not be relaxed. But different clocks can be used for different kinds of concerns. For example, compiler warnings should be addressed before commit but probably shouldn't require restarting a week-long vote. The point of the week is to give folks ample time to review a patch. Once concerns raised are addressed to the satisfaction of those who've raised them why would more time be required? However if someone asks for more time to review because the changes they requested are not easily reviewable in the time remaining then they should be given that time. Lastly, if syncing a branch with trunk is difficult as trunk changes, then there may be costs to delaying and we shouldn't delay without reason. Doug
Re: [DISCUSS] What is the purpose of merge vote threads?
On Fri, Oct 25, 2013 at 10:56 AM, Vinod Kumar Vavilapalli vino...@apache.org wrote: Discussing on a voting thread is not productive. When all votes are +1 then no discussion is needed. One shouldn't call a vote unless one expects all votes to be +1. But, if unexpectedly they're not all +1, then a discussion must ensue, to substantiate the veto and to try to establish a remedy. It seems overly formal to immediately terminate all votes at the first expression of concern, restarting them later. That puts process ahead of practicality and progress. Rather, if an unforeseen yet easily addressed concern is raised during a vote then folks might reasonably agree to continue without restarting the vote. The purpose of the vote is to establish consensus. If consensus is determined, then there's no need to delay. So a vote can pass when the -1 voters change their vote to +1. This might not hold if a remedy might be considered controversial, and its inclusion might reasonably invalidate prior +1 votes. Then more time might be given for folks to consider the remedy. But when the remedy is trivial it needn't be held to higher voting standard than a regular patch. Commits differ from releases since a release cannot be easily altered once published. However a commit can be amended by subsequent commits. We certainly want to minimize the need for subsequent commits, but don't need the same level of confidence. With branch merge votes we should focus on the issue of whether the project is ready to assume the burden of maintaining the new functionality, since it's much harder to remove things than add them. That's the reason for the one-week, 3 +1 rule. For minor issues like compiler warnings, a fix to a merge patch should be held to the same standard as any other patch. Doug
Re: [DISCUSS] What is the purpose of merge vote threads?
On Thu, Oct 24, 2013 at 2:51 PM, Chris Nauroth cnaur...@hortonworks.com wrote: When the voting happens on jira with a finalized merge patch, I know exactly what I'm voting for, because it's the same review-and-commit process that we follow every day with the extra requirement of 3 +1s. When it happens on email, it's less clear to me exactly what I'm voting for. Here's my take, FWIW. The entire project needs to determine whether it is willing to take on the maintenance of code developed in a branch. This vote needs the widest audience. On the other hand, discussion on the umbrella Jira for the branch concerns the precise details of the merge commit. Even if the project is generally willing to accept maintenance of the code in a branch, committing it should not break the build that day. So fine-tuning the precise details of the merge often happens in Jira rather than as a part of the formal merge vote. Sometimes these are determined in parallel. Since a -1 must always be accompanied by a rationale, it should be clear whether such a vote concerns a fine point of the specific patch that's easily corrected or whether it's about a fundamental problem with the branch that will take longer to address. Either sort might be cast in either place. A fine-point objection shouldn't reset voting clocks and a fundamental objection concerns things that cannot be fixed in a voting window. If something lies in the middle then should be discussion until consensus is reached about whether the merge date need be delayed, voting restarted later, etc. I don't see a need for a more rigid policy here since we haven't seen things running amok around branch merges due to a lack of policy. Is that consistent with other folks understanding? Doug
Re: git line endings trouble since recent trunk
http://www.apache.org/dev/version-control.html#https-svn-config Doug On Jun 28, 2013 1:03 PM, Colin McCabe cmcc...@alumni.cmu.edu wrote: Clarification: svn:eol-style = native causes the files to contain whatever the native platform used to check out the code uses. I think just setting this property on all the HTML files should resolve this and future problems. patch posted. C. On Fri, Jun 28, 2013 at 12:56 PM, Colin McCabe cmcc...@alumni.cmu.edu wrote: I think the fix for this is to set svn:eol-style to native on this file. It's set on many other files, just not on this one: cmccabe@keter:~/hadoopST/trunk svn propget svn:eol-style ./hadoop-project-dist/README.txt native cmccabe@keter:~/hadoopST/trunk svn propget svn:eol-style ./hadoop-hdfs-project/hadoop-hdfs/src/main/docs/releasenotes.html cmccabe@keter:~/hadoopST/trunk It might also be a good idea to run dos2unix on it. I thought that in general we wanted to have 'LF' everywhere, so perhaps we should add this to the patch.sh script to prevent this from re-occurring. Colin On Fri, Jun 28, 2013 at 12:27 PM, Sandy Ryza sandy.r...@cloudera.com wrote: I haven't been able to find a solution. Just filed https://issues.apache.org/jira/browse/HADOOP-9675. On Fri, Jun 28, 2013 at 12:19 PM, Omkar Joshi ojo...@hortonworks.com wrote: Sandy... did you fix this? any jira to track? me too facing same problem.. Thanks, Omkar Joshi *Hortonworks Inc.* http://www.hortonworks.com On Fri, Jun 28, 2013 at 11:54 AM, Zhijie Shen zs...@hortonworks.com wrote: Have the same problem here, have to edit the patch manually to exclude the changes in releasenotes.html On Fri, Jun 28, 2013 at 11:01 AM, Sandy Ryza sandy.r...@cloudera.com wrote: Has anybody else been having trouble with line endings since pulling trunk recently? hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html shows up as modified even though I haven't touched it, and I can't check it out or reset to a previous version to make that go away. The only thing I can do to neutralize it is to put it in a dummy commit, but I have to do this every time I switch branches or rebase. This appears to have began after the release notes commit (8c5676830bb176157b2dc28c48cd3dd0a9712741), and must be due to a line endings change. -Sandy -- Zhijie Shen Hortonworks Inc. http://hortonworks.com/
Re: [VOTE] - Release 2.0.5-beta
On Wed, May 15, 2013 at 10:57 AM, Arun C Murthy a...@hortonworks.com wrote: To get past all of this confusion, I'd like to present an alternate, specific proposal for consideration. I propose we continue the original plan and make a 2.0.5-beta release by May end with the following content: If you intend to nullify a prior vote then you should be very explicit, e.g., making that a clause in the new proposal. # nullify the result of the vote on general@ that started with message-id xxx... Also, why is this on common-dev? Isn't this list just for discussion of things that happen in the hadoop-common tree? http://hadoop.apache.org/mailing_lists.html#Common all attempts at calm, reasoned, civil discussion based on technical arguments have come to naught Wow. The folks you disagreed with there had absolutely no impact on your thinking about this release? Release numbers are cheap. We shouldn't fight over them. Doug
Re: [DISCUSS] create a hadoop-build subproject (a follow up on the thread 'introduce Python as build-time...')
On Wed, Dec 12, 2012 at 10:20 AM, Alejandro Abdelnur t...@cloudera.com wrote: you can do mvn install in the plugins project and then you'll be able to use it from the project you are using the plugin. Avro has its Maven plugins in a module that's used when compiling other modules. You can build all of Avro without installing any of it. So I do not think that Maven plugins need to be installed to be used, but can just be in a separate Maven module. The Hadoop plugins might, e.g., be placed a sub-module of Common. Doug
Re: [VOTE] introduce Python as build-time and run-time dependency for Hadoop and throughout Hadoop stack
On Mon, Dec 3, 2012 at 11:21 AM, Matt Foley mfo...@hortonworks.com wrote: It is intended to be a technical discussion, in the sense of the bylaws statement (in section Roles and Responsibilities: Committers), Committers may cast binding votes on any technical discussion regarding any subproject. I therefore intended it to be a majority vote of Committers. I'm not sure how you conclude that technical discussions are resolved with majority votes. http://www.apache.org/foundation/voting.html Interestingly, this need to discuss tooling and other issues that go beyond a simple code change is not addressed in the Decision Making: Actions section of the bylaws. That need seems to have been overlooked in the current rev of that section. But I do not agree that such issues are code changes; it relates to the tools we depend on to make code changes, which is clearly qualitatively different. I don't see a striking difference between this and a proposed code change. How is a -1 here fundamentally different than a veto on a patch submitted to HADOOP-9082? Doug
Re: [VOTE] introduce Python as build-time and run-time dependency for Hadoop and throughout Hadoop stack
Hadoop's bylaws do draw finer distinctions than the Apache voting guidelines document, but we follow the same general principles that are described there. As I understand it, the rationale for using consensus for code is that everyone needs to agree on everything in the codebase or we've disenfranchised some. We share a single code repository and we need to all agree on what goes into it. A release does not require majority since if someone doesn't agree on the timing of a release they can choose to make another at a different time, but every change that goes into each release requires consensus. We also require consensus for committers and PMC member votes so that we have a group that's coherent and is able to reach consensus on code changes. Re-writing bash scripts in Python is neither a release nor other procedural issue. It involves changes to the software we maintain and seems to fall clearly into the code change category. If you disagree then perhaps you'd like to propose a change to the bylaws so that scripts have different rules than other kinds of software, but I don't yet see the rationale for such a change. Doug On Mon, Dec 3, 2012 at 5:22 PM, Matt Foley ma...@apache.org wrote: No, but it speaks to whether the Hadoop bylaws can extend the Apache voting procedures and draw finer distinctions. For example, the Apache voting procedures only identify 3 types of votable issue, while the Hadoop bylaws identify 9 types of votable issues. If we were forced to fit development tools into one of the three categories cited by the Apache voting procedures, it would be fitting a square peg in a round hole. Since we can instead look at the 9 categories provided by the Hadoop bylaws, we can acknowledge that development tools was an overlooked category. But in my opinion it certainly doesn't fit into the code change category. Tooling is a meta-issue regarding HOW we do what needs to be done. In this case, whether we allow a platform-independent solution, or force contributors to maintain parallel scripts in multiple platform-specific languages for no reason. --Matt On Mon, Dec 3, 2012 at 3:57 PM, Doug Cutting cutt...@apache.org wrote: On Mon, Dec 3, 2012 at 2:08 PM, Matt Foley mfo...@hortonworks.com wrote: The apache voting process contradicts the Hadoop bylaws: http://www.apache.org/foundation/voting.html says that only PMC members can make binding votes on code modification issues, but http://hadoop.apache.org/bylaws.html says that Committers can make binding votes on them. Does that mean the Hadoop bylaws have to change? This may be a little atypical but I don't see any harm. The Hadoop PMC is willing to respect the veto of any committer as binding. I'd worry more if we tried to reduce vetoes to a subset of the PMC than extend it to a superset. Do you think this is problematic? Doug
Re: [VOTE] introduce Python as build-time and run-time dependency for Hadoop and throughout Hadoop stack
-1, +1, -1 Run- build-time scripting should be limited to operations that are impossible in Java. These should not be complex nor should we encourage more complexity in them. A parallel set of simple .bat files for such operations seems preferable to adding a Python dependency. Doug On Sat, Nov 24, 2012 at 12:13 PM, Matt Foley ma...@apache.org wrote: For discussion, please see previous thread [PROPOSAL] introduce Python as build-time and run-time dependency for Hadoop and throughout Hadoop stack. This vote consists of three separate items: 1. Contributors shall be allowed to use Python as a platform-independent scripting language for build-time tasks, and add Python as a build-time dependency. Please vote +1, 0, -1. 2. Contributors shall be encouraged to use Maven tasks in combination with either plug-ins or Groovy scripts to do cross-platform build-time tasks, even under ant in Hadoop-1. Please vote +1, 0, -1. 3. Contributors shall be allowed to use Python as a platform-independent scripting language for run-time tasks, and add Python as a run-time dependency. Please vote +1, 0, -1. Note that voting -1 on #1 and +1 on #2 essentially REQUIRES contributors to use Maven plug-ins or Groovy as the only means of cross-platform build-time tasks, or to simply continue using platform-dependent scripts as is being done today. Vote closes at 12:30pm PST on Saturday 1 December. - Personally, my vote is +1, +1, +1. I think #2 is preferable to #1, but still has many unknowns in it, and until those are worked out I don't want to delay moving to cross-platform scripts for build-time tasks. Best regards, --Matt
Re: Mailing list admin?
The moderators of this list can be contacted at common-dev-owner at hadoop.apache.org. If you'd like to become a moderator, please send a message to apmail at apache.org asking to become a moderator. CC private at hadoop.apache.org to keep the PMC in the loop. http://www.apache.org/dev/committers.html#mailing-list-moderators Doug On Wed, Nov 28, 2012 at 4:23 AM, Harsh J ha...@cloudera.com wrote: Is there no one amongst us who has list administrative rights? There's quite some problems affecting all users of the lists that needs to be addressed. Can I be granted rights (or at least be told how to get them), to do the moderation/administration work myself? On Wed, Oct 24, 2012 at 10:24 AM, Harsh J ha...@cloudera.com wrote: Ping? On Thu, Oct 18, 2012 at 5:25 PM, Harsh J ha...@cloudera.com wrote: Hey project devs, Can someone let me know who the MLs admin is? INFRA suggested that instead of going to them, I could reach out to the admin group local to the project itself (I didn't know we had admins locally). P.s. Happy to volunteer to administrate as well. Please ping me directly, Thanks, Harsh J -- Harsh J -- Harsh J
[jira] [Resolved] (HADOOP-8662) remove separate pages for Common, HDFS MR projects
[ https://issues.apache.org/jira/browse/HADOOP-8662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Doug Cutting resolved HADOOP-8662. -- Resolution: Fixed I committed this. The new site is now live. Konstantin: I removed your affiliation from the who.html page. Chris: I updated the affiliations you listed. remove separate pages for Common, HDFS MR projects Key: HADOOP-8662 URL: https://issues.apache.org/jira/browse/HADOOP-8662 Project: Hadoop Common Issue Type: Improvement Components: documentation Affects Versions: site Reporter: Doug Cutting Assignee: Doug Cutting Priority: Minor Fix For: site Attachments: HADOOP-8662.patch, HADOOP-8662.patch The tabs on the top of http://hadoop.apache.org/ link to separate sites for Common, HDFS and MapReduce modules. These sites are identical except for the mailing lists. I propose we move the mailing list information to the TLP mailing list page and remove these sub-project websites. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-8752) Update website to reflect merged committer lists
[ https://issues.apache.org/jira/browse/HADOOP-8752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Doug Cutting resolved HADOOP-8752. -- Resolution: Duplicate Fix Version/s: site Assignee: (was: Eli Collins) This was subsumed by HADOOP-8662. Update website to reflect merged committer lists Key: HADOOP-8752 URL: https://issues.apache.org/jira/browse/HADOOP-8752 Project: Hadoop Common Issue Type: Task Reporter: Eli Collins Priority: Minor Fix For: site Let's update the website to reflect the merged committer list (http://s.apache.org/Owx), ie one top-level credits section next to the PMC section. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8662) remove separate pages for Common, HDFS MR projects
Doug Cutting created HADOOP-8662: Summary: remove separate pages for Common, HDFS MR projects Key: HADOOP-8662 URL: https://issues.apache.org/jira/browse/HADOOP-8662 Project: Hadoop Common Issue Type: Improvement Components: documentation Affects Versions: site Reporter: Doug Cutting Assignee: Doug Cutting Priority: Minor Fix For: site The tabs on the top of http://hadoop.apache.org/ link to separate sites for Common, HDFS and MapReduce modules. These sites are identical except for the mailing lists. I propose we move the mailing list information to the TLP mailing list page and remove these sub-project websites. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: MultithreadedMapper
On Thu, Jul 26, 2012 at 7:42 AM, Robert Evans ev...@yahoo-inc.com wrote: About the only time that MultiThreaded mapper makes a lot of since is if there is a lot of computation associated with each key/value pair. Or if the mapper does a lot of i/o to some external resource, e.g., a web crawler. Doug
Re: [RESULT] - [VOTE] Rename hadoop branches post hadoop-1.x
On 03/28/2012 12:39 PM, Owen O'Malley wrote: [ ... ] So the RM of the 2 branch needs to make the call of what should be 2.1 vs 3.0. I thought these were community decisions, not RM decisions, no? http://s.apache.org/rm Doug
Re: Apache Jenkins is down
On 02/12/2012 10:02 PM, Roman Shaposhnik wrote: For issues like these it always helps to file an INFRA Jira. It's good to first check the ASF monitoring status page: http://monitoring.apache.org/status/ If there's a red on that page then Infrastructure is already aware of the problem. Another thing is to listen in on #asfinfra on irc.freenode.net. Doug
[jira] [Reopened] (HADOOP-7920) Remove Avro RPC
[ https://issues.apache.org/jira/browse/HADOOP-7920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Doug Cutting reopened HADOOP-7920: -- There are a few more things that can be removed. Remove Avro RPC --- Key: HADOOP-7920 URL: https://issues.apache.org/jira/browse/HADOOP-7920 Project: Hadoop Common Issue Type: Bug Components: ipc Affects Versions: 0.23.1 Reporter: Suresh Srinivas Assignee: Suresh Srinivas Fix For: 0.24.0, 0.23.1 Attachments: HADOOP-7920.patch, HADOOP-7920.patch, HADOOP-7920.txt Please see the discussion in HDFS-2660 for more details. I have created a branch HADOOP-6659 to save the Avro work, if in the future some one wants to use the work that existed to add support for Avro RPC. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Dropping CHANGES.txt?
On 11/21/2011 11:49 AM, Matt Foley wrote: The advantage of CHANGES.txt is that it merges when the branch merges. So as long as the developers stick to reasonably normal merge and commit processes, it truly reflects what is in the current state of any given branch, at least at the level of changes related to Jiras. I agree. Its redundancy makes it valuable. I often use CHANGES.txt diffs to double-check that I've merged the right revision to a branch. Doug
[jira] [Created] (HADOOP-7693) fix RPC.Server#addProtocol to work in AvroRpcEngine
fix RPC.Server#addProtocol to work in AvroRpcEngine --- Key: HADOOP-7693 URL: https://issues.apache.org/jira/browse/HADOOP-7693 Project: Hadoop Common Issue Type: Improvement Components: ipc Reporter: Doug Cutting Assignee: Doug Cutting HADOOP-7524 introduced a new way of passing protocols to RPC servers, but it was not implemented correctly by AvroRpcEngine in that issue. This is required to fix HDFS-2298. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-4905) static initializers for default config files duplicate code
[ https://issues.apache.org/jira/browse/HADOOP-4905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Doug Cutting resolved HADOOP-4905. -- Resolution: Invalid Resolving as 'Invalid' since these static initializers are no long present in the codebase. static initializers for default config files duplicate code --- Key: HADOOP-4905 URL: https://issues.apache.org/jira/browse/HADOOP-4905 Project: Hadoop Common Issue Type: Improvement Components: conf Affects Versions: 0.20.0 Reporter: Doug Cutting Priority: Minor Labels: newbie The default config files are loaded by static initializers. The code in these initializers is two lines that contains string literals. This is fragile and duplicated code. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: JIRA attachments order
On 09/10/2011 01:32 AM, Vinod Kumar Vavilapalli wrote: Regarding patch names, I too agree with Allen that extension should be .txt. I do run into patches every now and then which aren't interpreted as text files, so there are people out there using browsers that don't properly set the content-type. I have seen .txt attachments that were not uploaded as text. A better thing might be to ask folks to check the mime type they've uploaded, as displayed in Jira, to make sure it's text. What I do to remove variability is not click on the patch directly, but always use right-click and choose Save Link As (or your browser's equivalent). Doug
Re: JIRA attachments order
On 09/09/2011 07:27 AM, Ted Dunning wrote: If you post the same patch with the same name, JIRA helps you out by greying all the earlier versions out. Indeed. That's the best practice, not to add version numbers to patch files, for this very reason. We should perhaps note this on: http://wiki.apache.org/hadoop/HowToContribute I am a Jira administrator and would be happy to change the default ordering of attachments if it were possible, however I can see no option to do so. Doug
Re: JIRA attachments order
On 09/09/2011 11:12 AM, Eli Collins wrote: Personally I like version numbers as well, it allows me to refer to a specific version of the patch (vs a patch on a given time of date/date). Re-using the name doesn't hide the old versions, it just makes them gray. They're still listed, with date and may be sorted by date. If you select the ActivityAll tab then the different versions are linked to in the comment stream, providing context. 90+% of the time I'm interested in the most recent version of the patch, so the value of having it highlighted is great. Frequently when different names are used I mistakenly download the wrong version and waste time reviewing it, but when the same name is used I always get the most recent. The highlighting is very effective for me. Doug
Re: JIRA attachments order
On 09/09/2011 01:38 PM, Kirby Bohling wrote: Someday I wish Apache would find/adopt a distributed version control system (I know about git.apache.org, but I mean pushing that further), and use something like gerrit or review board. So if you have a patch, or a series of patches, you'd just use a version control system. Make it such that when you submit to review system you have to log in and say Yes, I want contribute this under to the ASF under an Apache 2.0 license. Make me use my JIRA/Apache credentials to submit reviews directly from my version control, and force the commit message to be properly formatted to tie it to a JIRA issue. Make the patch submission have a link right next to it to take me to the review system, where it can be +1/0/-1'ed with commentary in line. The appropriate forum to discuss that is infrastructure-dev@a.o. http://www.apache.org/dev/infra-mail.html In the meantime, we should use the tools we have as best we can. Doug
[jira] [Resolved] (HADOOP-7562) Update asf-mailer.conf for new trees
[ https://issues.apache.org/jira/browse/HADOOP-7562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Doug Cutting resolved HADOOP-7562. -- Resolution: Fixed Assignee: Doug Cutting Hadoop Flags: [Reviewed] Turns out committers cannot see this file. PMC chairs are meant to maintain it. I committed this. Should do the right thing now. Update asf-mailer.conf for new trees Key: HADOOP-7562 URL: https://issues.apache.org/jira/browse/HADOOP-7562 Project: Hadoop Common Issue Type: Bug Reporter: Todd Lipcon Assignee: Doug Cutting Priority: Critical Attachments: HADOOP-7562.patch Since the mavenization, we have renamed various directories inside hadoop/trunk. The asf-mailer.conf file in the infrastructure SVN needs to be updated for these new paths in order to fix the commit mails and the triggering of the git mirror. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [VOTE] Release candidate 0.20.203.0-rc0
On 04/29/2011 04:09 PM, Owen O'Malley wrote: I think everything is ready to go on the 0.20.203.0 release. It includes security and a lot of improvements in the capacity scheduler and JobTracker. This does not appear to include the 0.20-append work? So it's not advisable to use HBase with this revision, correct? Should we release http://people.apache.org/~omalley/hadoop-0.20.203.0-rc0/? The patch selection process for this branch did not appear to be a community process. A massive patch set was committed en-masse with no public discussion before or after about its specific composition. Long-term the users of a project benefit from a community that collaborates using open, interactive processes. If a particular set of patches, not created through such a process, is valuable to end users, then it can be distributed on github or elsewhere under a different name, but should not be granted the imprimatur of a community product. Doug
Re: [VOTE] Release candidate 0.20.203.0-rc0
On 05/02/2011 11:37 AM, Alan Gates wrote: From the viewpoint of a downstream user, I'd like to see this released. Right now Hive 0.7 and soon HCatalog 0.1 have to depend on a Cloudera distribution because they need security. Having Apache products depend on 3rd party distributions of Apache products is bogus. The sooner this is out the sooner we can fix this. Alan, Cloudera could upload its CDH3 patchset to a branch in Apache subversion and call a release vote on it and I would vote against it. The interactive community process is to me what makes it Apache. Releases should branch from trunk or use an existing release branch. A release branch should be open for patches from the general community. Neither were the case here. This is neither a subset or a superset of the 0.20 branch that the community has invested in. The change log for this includes around 500 changes, yet only 24 issues are assigned to it in Jira, the community's issue tracker. Yes, the current situation is bad, but shortcutting the community process doesn't fix it, it just hides it. Cheers, Doug
Re: [Hadoop Wiki] Update of FrontPage by prosch
I edited the LocalBadContent page to prohibit linking to these domains. Doug On 01/10/2011 02:46 AM, Niels Basjes wrote: Seems like a good moment to blacklist the prosch user from ever changing the wiki again. 2011/1/10 Apache Wikiwikidi...@apache.org: Dear Wiki user, You have subscribed to a wiki page or wiki category on Hadoop Wiki for change notification. The FrontPage page has been changed by prosch. http://wiki.apache.org/hadoop/FrontPage?action=diffrev1=175rev2=176 -- == General Information == * [[http://hadoop.apache.org/|Official Apache Hadoop Website]]: download, bug-tracking, mailing-lists, etc. * [[ProjectDescription|Overview]] of Apache Hadoop - * [[FAQ]] + * [[FAQ]] [[http://www.profi-fachuebersetzung.de/language-translation.html|Translation agency]] / [[http://www.profischnell.com|Übersetzung Polnisch Deutsch]] * [[HadoopIsNot|What Hadoop is not]] * [[Distributions and Commercial Support|Distributions and Commercial Support]] for Hadoop (RPMs, Debs, AMIs, etc) * [[HadoopPresentations|Presentations]], [[Books|books]], [[HadoopArticles|articles]] and [[Papers|papers]] about Hadoop * PoweredBy, a list of sites and applications powered by Apache Hadoop * Support * [[Help|Getting help from the hadoop community]]. - * [[Support|People and companies for hire]]. + * [[Support|People and companies for hire]]. * [[Conferences|Hadoop Community Events and Conferences]] * HadoopUserGroups (HUGs) * HadoopSummit
Re: Jira workflow problem.
The problem was that the submitter could transition from Patch Available to In Progress, following the Resume Progress transition, but the submitter cannot then transtion anywhere from In Progress, only the assignee could. I fixed that, so that the assignee can no longer follow that transition to a cul de sac. I also made you a contributor so you could be assigned issues, and assigned you this issue. Doug On 01/06/2011 01:41 PM, Niels Basjes wrote: I seem to have selected the wrong option in Jira to get the latest patch handled. For some reason the option to indicate a new patch has been made available is no longer present. https://issues.apache.org/jira/browse/HADOOP-7076 What did I do wrong and what can I do to fix this? Thanks.
[jira] Resolved: (HADOOP-4617) hadoop-default.xml should not be in conf/ directory
[ https://issues.apache.org/jira/browse/HADOOP-4617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Doug Cutting resolved HADOOP-4617. -- Resolution: Duplicate Thanks for noticing this stale issue, Paul! what about hdfs-default and mapred-default documentation? Probably new issues should be filed if that documentation still needs to be updated. hadoop-default.xml should not be in conf/ directory --- Key: HADOOP-4617 URL: https://issues.apache.org/jira/browse/HADOOP-4617 Project: Hadoop Common Issue Type: Improvement Components: conf Reporter: Doug Cutting It is error prone to keep hadoop-default.xml in the conf/ directory. Folks copy configuration directories between releases, but this file is strongly tied to a particualr release and should not be either editted or used with other than the release it comes with. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HADOOP-7020) establish a Powered by Hadoop logo
establish a Powered by Hadoop logo Key: HADOOP-7020 URL: https://issues.apache.org/jira/browse/HADOOP-7020 Project: Hadoop Common Issue Type: Improvement Components: documentation Affects Versions: site Reporter: Doug Cutting Fix For: site We should agree on a Powered By Hadoop logo, as suggested in: http://www.apache.org/foundation/marks/pmcs#poweredby -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[HADOOP-6317] Serialize the 'final'ness of Configuration keys - ASF JIRA
Arun, are you still interested in this issue? https://issues.apache.org/jira/browse/HADOOP-6317 This was the oldest item in the patch queue. I have updated the patch to apply to current trunk. It looks like a reasonable feature to me. You originally filed the issue. Aaron supplied a patch about a year ago. Thoughts? Thanks, Doug
[jira] Resolved: (HADOOP-6992) Update website for recent subproject departures
[ https://issues.apache.org/jira/browse/HADOOP-6992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Doug Cutting resolved HADOOP-6992. -- Resolution: Fixed Hadoop Flags: [Reviewed] Hive has now moved its website, so I have now committed this. Update website for recent subproject departures --- Key: HADOOP-6992 URL: https://issues.apache.org/jira/browse/HADOOP-6992 Project: Hadoop Common Issue Type: Improvement Components: documentation Affects Versions: site Reporter: Doug Cutting Assignee: Doug Cutting Fix For: site Attachments: HADOOP-6992.patch, hadoop-tlp.pdf A number of subprojects have left Hadoop yet the website's not been fully updated to reflect that. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
Re: Page BayAreaHadoopUserGroup deleted from Hadoop Wiki
Does anyone know who SomeOtherAccount is? They've been making good changes to the wiki. It'd be nice to know who this is, rather than keeping it anonymous. Thanks, Doug On 09/17/2010 09:59 AM, Apache Wiki wrote: Dear wiki user, You have subscribed to a wiki page Hadoop Wiki for change notification. The page BayAreaHadoopUserGroup has been deleted by SomeOtherAccount. The comment on this change is: It is wrong, completely out of date, never updated, and replaced with a link to http://www.meetup.com/hadoop/ anyway. http://wiki.apache.org/hadoop/BayAreaHadoopUserGroup
Re: Plans for a 0.21 Hadoop Release
Owen O'Malley wrote: and hopefully to get the new generic serialization api in before the branch. Is this just HADOOP-6685? Or does it also include MAPREDUCE-1183? My instinct would rather be to revert the serialization API changes made since 0.20 (mostly HADOOP-6165) if that's not an API we intend to support, and not add new APIs at the last minute. Tom's proposed that the serialization API is evolving, so we can technically change it going forward, but we should still be cautious about promoting new APIs without a clear consensus, and serialization APIs have proven controversial in the recent past. It might be also better to get some experience with a new API in trunk before we include it in a release. Doug
Re: [DISCUSSION] Release process
Owen O'Malley wrote: In my experience with releasing Hadoop, the bare minimum of scale testing is a couple of weeks on 500 nodes (and more is far better) with a team of people testing it. I think that releasing a 1.0 that has never been tested at scale would be disastrous. For the record, I never proposed getting a 1.0 release out that had been scale tested in a few weeks. Rather, I proposed getting an alpha 1.0 release out in a few weeks. Bugfix releases from that, after testing, could then be made. My claim was that we might sooner get a stable Apache release that supports append via this route than from trunk. But, since there is opposition to this proposal, I will abandon it. Doug
Re: [DISCUSSION] Release process
Chris Douglas wrote: Speaking of the release vote process, I renew my request that we formalize both the RM role and the bylaws. -C I think the HTTPD release rules are non-controversial and would support adoption of something similar. Someone needs to draft a proposal, initiate a discussion, refine the draft, and finally vote to enact it. It's kind of like a release, in that its best if someone manages the process. Would you like to do this? Similarly, for bylaws, we need someone to lead the process. The previous attempt started with a vote, rather than a discussion. That vote turned into a discussion that never turned back into a vote. Doug
Re: [DISCUSSION] Release process
Chris K Wensel wrote: are we saying we will de-deprecate the stable APIs in .20, or make the new APIs introduced in .20 stable? +1 on removing the deprecations on the stable APIs. Yes. I too am +1 on removing deprecations in stable, public APIs in a 1.0 release. Code that uses only public 1.0 APIs and compiles without deprecation warnings against 1.0 is code we should support going forward. Doug
Re: [DISCUSSION] Release process
Todd Lipcon wrote: With HDFS-200 we'd also need HDFS-142 Good to know. I' have to admit to being puzzled by HDFS-200, since Nicholas resolved it as a duplicate on 7 January, yet Dhruba's continued to post patches to it. Dhruba, Stack: do you have any thoughts on the appropriateness of making a release with HDFS-200 HDFS-142? and potentially other fixes yet to be determined (this append work based on 200 is still ongoing). I don't think it will be stable release quality within a few weeks. I assume that HDFS-200 as-is does more good than harm, no? Also, 1.0.0 doesn't need to be flawless. If we identify critical bugs after its release, then we'll make a 1.0.1 release. We might even call the first 1.0 release something like 1.0.0 alpha. That said, I do believe it will still be stable sooner than the release from trunk. Doug
Re: [DISCUSSION] Release process
Owen O'Malley wrote: It is tempting and I think that 0.20 is *really* our 1.0, but I think re-labeling a release a year after it came out would be confusing. I wasn't proposing just a re-labeling. I was proposing a new release, branched from 0.20 rather than trunk. We'd introduce some changes, after voting on each of course. Candidates are MAPREDUCE-1623 and MAPREDUCE-1650, to better clarify what's intended to be supported in 1.0, and HDFS-200, to make append reliable. Since we have not yet made a 0.21 release, this numbering would be consistent. It also naturally permits further 1.x releases that add features, like security. Doug
Re: [DISCUSSION] Release process
Konstantin Shvachko wrote: I would like to propose a straightforward release of 0.21 from current 0.21 branch. That could be done too. Tom's volunteered to drive a release from trunk in a few weeks. Owen's volunteered to drive another release from trunk in about six months. Would you like to volunteer to drive a release from the current 0.21 branch? My latest proposal, a 1.0 branch based on 0.20, contains two questions: 1. Should we make an Apache release that more closely corresponds to what folks are using in production today (and will be using for a while yet)? 2. If we're considering the 0.20 mapreduce and filesystem APIs to be 1.0 APIs, and the new mapreduce and filesystem APIs to be 2.0 APIs, shouldn't our release numbering reflect that? Release numbers are fundamentally about compatibility declarations. We could instead change our compatibility rules to specifically mention certain release numbers, but that feels the wrong way around. Since we've not yet made a 0.21 release, we still have an opportunity to interject a 1.0 release with the semantics folks expect: its public APIs are stable. If there's support for this proposal, then I'd volunteer to drive it. I won't bother to pursue this unless folks think it is worthwhile, so, if you like it, please speak up. While a release itself cannot be vetoed and only requires a simple majority, we'll need to vote which patches would be applied to a 1.0 branch, and those code-change votes require consensus, so, vetos there would stop the process. So please also speak up if you strongly oppose this proposal. Doug
[jira] Resolved: (HADOOP-6660) add wrapper for Avro data
[ https://issues.apache.org/jira/browse/HADOOP-6660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Doug Cutting resolved HADOOP-6660. -- Resolution: Duplicate Fix Version/s: (was: 0.22.0) add wrapper for Avro data - Key: HADOOP-6660 URL: https://issues.apache.org/jira/browse/HADOOP-6660 Project: Hadoop Common Issue Type: New Feature Components: io Reporter: Doug Cutting To permit passing Avro data through MapReduce we can add a wrapper class and serialization for it. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
Re: [DISCUSSION] Release process
Tom White wrote: I think the focus should be on getting an alpha release out, so I suggest we create a new 0.21 branch from trunk Another release we might consider is 1.0 based on 0.20. We'd then have releases that correspond to what folks are actually using in production. This would also rationalize our release numbering, since many have expressed that 0.20 APIs should be treated as 1.0 APIs. A 1.0 release based off 0.20 would give us a chance to state more precisely the 1.0 API that we intend to support long-term. For example, we might un-mark the old mapreduce APIs as deprecated in a 1.0 release, and mark the new mapreduce APIs as experimental and unstable there. Programs that use only public stable features in 1.0 could be then guaranteed to run for a long-time hence. It would also be good to get HDFS-200 into 1.0. That might be the fastest route to providing a stable append for HBase. Y!'s 0.20+security could become the basis of a 1.1 release. The next release from trunk might then be called 2.0 alpha. It would support 1.0 APIs, but they'd be deprecated in favor of newer API for mapreduce and filesystems. We could pursue releasing 1.0 and 2.0 alpha in parallel. Thoughts? Doug
[jira] Created: (HADOOP-6660) add Writable wrapper for Avro data
add Writable wrapper for Avro data -- Key: HADOOP-6660 URL: https://issues.apache.org/jira/browse/HADOOP-6660 Project: Hadoop Common Issue Type: New Feature Components: io Reporter: Doug Cutting Fix For: 0.22.0 To permit passing Avro data through MapReduce we can add a wrapper class that implements Writable. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HADOOP-6659) Switch RPC to use Avro
Switch RPC to use Avro -- Key: HADOOP-6659 URL: https://issues.apache.org/jira/browse/HADOOP-6659 Project: Hadoop Common Issue Type: Improvement Components: ipc Reporter: Doug Cutting This is an umbrella issue for moving HDFS and MapReduce RPC to use Avro. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HADOOP-6629) versions of dependencies should be specified in a single place
versions of dependencies should be specified in a single place -- Key: HADOOP-6629 URL: https://issues.apache.org/jira/browse/HADOOP-6629 Project: Hadoop Common Issue Type: Improvement Components: build Reporter: Doug Cutting Currently the Maven POM file is generated from a template file that includes the versions of all the libraries we depend on. The versions of these libraries are also present in ivy/libraries.properties, so that, when a library is updated, it must be updated in two places, which is error-prone. We should instead only specify library versions in a single place. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
Re: JIRA Edits?
Owen O'Malley wrote: Doug decided he didn't like the majority of people editing their comments on jira and disabled edits. I did propose that we disable comment editing. There was some discussion, but no one vetoed the proposal, so I implemented it. The rationale is that comment edits as implemented by Jira make it hard to follow an issue. If you're following an issue by email, and after you read a 3-paragraph comment, it's then edited, you are re-sent the entire edited comment, not a diff, and must attempt to figure out what's changed: whether it's just a spelling correction or whether it's a more substantial change. Things are no better in the web interface, since, if one has already read the comment and then it's edited, one must re-read the entire message to determine what's changed. With many people following each issue, edits incur a cost on the community. Following Hadoop Jira issues already consumes a lot of our collective time, and edits increase this. Rather folks can consider Jira comments like email and IRC messages. One should review before one posts, and, if one makes a mistake, one should amend it with a follow-up posting. This makes it easier for folks to follow issues either via email and via the web interface. That said, if most folks now find this objectionable, we can hold a vote to consider changing it. Doug
[jira] Created: (HADOOP-6486) fix common classes to work with Avro 1.3 reflection
fix common classes to work with Avro 1.3 reflection --- Key: HADOOP-6486 URL: https://issues.apache.org/jira/browse/HADOOP-6486 Project: Hadoop Common Issue Type: Improvement Components: ipc Reporter: Doug Cutting Assignee: Doug Cutting Fix For: 0.22.0 A few minor changes are required to get some common classes to work correctly with Avro 1.3 reflection. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HADOOP-6422) permit RPC protocols to be implemented by Avro
permit RPC protocols to be implemented by Avro -- Key: HADOOP-6422 URL: https://issues.apache.org/jira/browse/HADOOP-6422 Project: Hadoop Common Issue Type: New Feature Components: ipc Reporter: Doug Cutting Assignee: Doug Cutting Fix For: 0.22.0 To more easily permit Hadoop to evolve to use Avro RPC, I propose to change RPC to use different implementations for clients and servers based on the configuration. This is not intended as an end-user configuration: only a single RPC implementation will be supported in a given release, but rather a tool to permit us to more easily develop and test new RPC implementations. As such, the configuration parameters used would not be documented. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
Re: Hadoop coding style guideline
Aaron Kimball wrote:
First, I've been picked on by others for using this brace style:
if (foo) {
stmt;
} else {
otherstmt;
}
and have been told to drop the braces because they look ugly if stmt or
otherstmt are only one line.
In http://java.sun.com/docs/codeconv/html/CodeConventions.doc6.html#449though,
the sun coding conventions *clearly* say that braces are always to
be used. Can we get a ruling here?
My preference is to permit both. I like to maximize the amount of
readable logic per screen, and find that close braces around one-line
expressions don't improve readability (since indentation already
indicates the nesting) and decrease the amount of per-screen logic.
However I know reasonable people who prefer to always fully-bracket
their code. I see no strong reason to force one style over the other.
And second, what's our story on tabs vs. spaces?
Spaces are preferred, since tabs are inconsistently interpreted. The
correct interpretation of a tab is to move to the next column evenly
divisible by 8, but many editors are configured differently, so tabs are
best avoided.
Doug
HADOOP-6318: upgrade to Avro 1.2.0
Can another committer please review this issue? https://issues.apache.org/jira/browse/HADOOP-6318 Thanks, Doug
[jira] Created: (HADOOP-6318) Upgrade to Avro 1.2.0
Upgrade to Avro 1.2.0 - Key: HADOOP-6318 URL: https://issues.apache.org/jira/browse/HADOOP-6318 Project: Hadoop Common Issue Type: Improvement Components: io, ipc Reporter: Doug Cutting Assignee: Doug Cutting Avro 1.2 has been released. The API's Hadoop Common uses have been simplified, and it should be upgraded. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HADOOP-6267) build-contrib.xml unnecessarily enforces that contrib projects be located in contrib/ dir
[
https://issues.apache.org/jira/browse/HADOOP-6267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Doug Cutting resolved HADOOP-6267.
--
Resolution: Fixed
Fix Version/s: 0.21.0
I just committed this. Thanks, Todd!
build-contrib.xml unnecessarily enforces that contrib projects be located in
contrib/ dir
-
Key: HADOOP-6267
URL: https://issues.apache.org/jira/browse/HADOOP-6267
Project: Hadoop Common
Issue Type: Improvement
Components: build
Reporter: Todd Lipcon
Assignee: Todd Lipcon
Priority: Minor
Fix For: 0.21.0
Attachments: hadoop-6267.txt
build-contrib.xml currently sets hadoop.root to ${basedir}/../../../. This
path is relative to the contrib project which is assumed to be inside
src/contrib/. We occasionally work on contrib projects in other repositories
until they're ready to contribute. We can use the dirname ant task to do
this more correctly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
Re: Towards Hadoop 1.0: Stronger API Compatibility from 0.21 onwards
Sanjay Radia wrote: No. The 1.0 proposal was that it included both API and wire compatibility. The proposal includes a lot of things, but it's so far just a proposal. There's been no vote to formally define what 1.0 will mean. In every discussion I've heard, from the very beginning of the project, it primarily meant API stability. You've added wire compatibility, data stability, security, restart recovery, etc. These are all very nice features to have, essential perhaps in some contexts, but they may nor may not be required for 1.0. I worry that if we keep piling more things on, we'll never get to 1.0. What would be wrong with calling it 1.0 when we have end-user API stability? Why would that be a bad thing? Doug
[jira] Resolved: (HADOOP-6190) Links to Hadoop mailing list mbox files are broken
[ https://issues.apache.org/jira/browse/HADOOP-6190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Doug Cutting resolved HADOOP-6190. -- Resolution: Fixed Assignee: Doug Cutting Good idea. I created that link. Thanks for noticing this, Tom. Links to Hadoop mailing list mbox files are broken -- Key: HADOOP-6190 URL: https://issues.apache.org/jira/browse/HADOOP-6190 Project: Hadoop Common Issue Type: Bug Reporter: Tom White Assignee: Doug Cutting Fix For: site http://hadoop.apache.org/mail/ returns a 404 (compare, for example, http://lucene.apache.org/mail/). There should be a symbolic link in /www/hadoop.apache.org/ from mail to /home/apmail/public-arch/hadoop.apache.org on people.apache.org. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
