Re: Project Workflow
Dear all, Finally I read all of this thread. Now I am writing to express my sincere apologies for my words and failure to control my anger. I realized that some of my behavior inappropriate and disrespectful. I hope that you will allow me the opportunity to express my apology again in person this Thursday. I will appreciate if someone can answer the following two questions. - How can I commit to the Git repo at the ASF directly? Is this what I should do? What makes this push different from Github pull request? $ cd climate $ git remote add apache https://git-wip-us.apache.org/repos/asf/climate.git $ git push apache $branch - What is canonical source code? There are also some silly questions. What do RTC, CTR and PMC stand for? Without these abbreviations, it would be much easier to understand some of email. Your sincerely, Kyo On 5/12/15, 10:36 PM, Cameron Goodale good...@apache.org wrote: Wow! There sure is a lot of passion and fire in this thread, so let me put on my asbestos small clothes and wade into the pool. Dear Kyo, When you say that this discussion is a waste of your time it makes me sad. Sure sitting here the last 20 minutes reading this thread could be seen as a waste of my time too, but I did it and I am writing this because I care about my friends on this project, including you. I really hope you were writing out of frustration earlier, and see that this isn't a waste of your time. To everyone else that reads this, My sincere hope is that everyone on this project wants what is best for the project, best community, best climate tools, best code, best version control, etc... so when tempers flare and people are using ALL CAPS, please try to slow it down a notch and try to see it from the other person's perspective. Thanks for reading, Cameron On Tue, May 12, 2015 at 9:57 PM, Michael Joyce jo...@apache.org wrote: Thank you for the great responses Lewis. You put into words what I haven't seemed to be able to type out today. Few comments below (it's really mostly just me typing +100 to be honest though) -- Jimmy On Tue, May 12, 2015 at 8:29 PM, Lewis John Mcgibbney lewis.mcgibb...@gmail.com wrote: Hi Chris, Please see replies inline On Tue, May 12, 2015 at 6:10 PM, dev-digest-h...@climate.apache.org wrote: I¹m honestly not sure what you are talking about, Not even one wee bit ;) maybe some of my replies will bridge the gap between you not being sure about what I am talking about Vs disagreeing with what I am trying to say. and to be honest, waiting 72 hours before committing everything is not a project I¹d ever want to be on. Well lets for a minute consider the flip side (or another possible side) of this coin. I would never want to be on a project where patches are merged which clearly break 50% of the tests in the entire codebase. It seems to me, rather detrimental to the project codebase as well as other community members who have confidence in a codebase for patches to be committed and merged whcih essentially break everything! Additionally, this goes against all of the good practice I've ever learned since stumbling across TheASF some 6 years ago! I can't word this opinion any other way. I'll add a +1 to not being interested on working on a project where the build is broken all the time because people can't be bothered to run the test suite. It wastes a huge amount of limited developer resource tracking down bugs for people. If you don't know how to run the tests, just ask. It's literally 1 command to run the entire test suite for the OCW package. If you're not sure just ask and people will help you out!! I don't know a single person here who wouldn't be willing to offer guidance and I know I have personally offered up my time to help out more times than I can count. Like I said - if I care about a review, or want something to be seen by someone, fine, I can choose to ask for it. Absolutely correct. I agree. It shouldn¹t be *imposed on me*. It seems like imposed is a strong word given the sentiment of the thread and the openness of Mike to open it up initially to how people want to do things. I think what we are trying to determine here is whether people feel like things are being imposed upon them. If that ends up being one of the outcomes of this thread then we need to accept, address, change, implement and move on. I consider a grace period as a politeness as well as a duration which people can gauge the contribution(s) and comment accordingly. That is all. It is not so people can shoot it down. That would be detrimental indeed. Imposed is way too strong of a word in my opinion. This is the workflow the project has laid out and used for the last year. I will agree we need better documentation on it (like this is the only thing). The intention of this thread is to make sure it's working appropriately for the project.
Re: Project Workflow
Good Morning Kyo, I am sorry for your frustration in this matter, and I appreciate you sticking it out with us. I'll let the git professionals answer the code related stuff and I'll handle the acronyms. CTR - Commit then review ( doing code review after it has been committed) RTC - Review then Commit ( waiting for a review of the code, then committing it to the repo) PMC - Project Management Committee (http://www.apache.org/dev/pmc.html) I admit it took me about 2 or 3 emails before CTR and RTC became clear to me. Best Regards, Cam On Wed, May 13, 2015 at 3:54 AM, Lee, Kyo (329C-Caltech) huikyo@jpl.nasa.gov wrote: Dear all, Finally I read all of this thread. Now I am writing to express my sincere apologies for my words and failure to control my anger. I realized that some of my behavior inappropriate and disrespectful. I hope that you will allow me the opportunity to express my apology again in person this Thursday. I will appreciate if someone can answer the following two questions. - How can I commit to the Git repo at the ASF directly? Is this what I should do? What makes this push different from Github pull request? $ cd climate $ git remote add apache https://git-wip-us.apache.org/repos/asf/climate.git $ git push apache $branch - What is canonical source code? There are also some silly questions. What do RTC, CTR and PMC stand for? Without these abbreviations, it would be much easier to understand some of email. Your sincerely, Kyo On 5/12/15, 10:36 PM, Cameron Goodale good...@apache.org wrote: Wow! There sure is a lot of passion and fire in this thread, so let me put on my asbestos small clothes and wade into the pool. Dear Kyo, When you say that this discussion is a waste of your time it makes me sad. Sure sitting here the last 20 minutes reading this thread could be seen as a waste of my time too, but I did it and I am writing this because I care about my friends on this project, including you. I really hope you were writing out of frustration earlier, and see that this isn't a waste of your time. To everyone else that reads this, My sincere hope is that everyone on this project wants what is best for the project, best community, best climate tools, best code, best version control, etc... so when tempers flare and people are using ALL CAPS, please try to slow it down a notch and try to see it from the other person's perspective. Thanks for reading, Cameron On Tue, May 12, 2015 at 9:57 PM, Michael Joyce jo...@apache.org wrote: Thank you for the great responses Lewis. You put into words what I haven't seemed to be able to type out today. Few comments below (it's really mostly just me typing +100 to be honest though) -- Jimmy On Tue, May 12, 2015 at 8:29 PM, Lewis John Mcgibbney lewis.mcgibb...@gmail.com wrote: Hi Chris, Please see replies inline On Tue, May 12, 2015 at 6:10 PM, dev-digest-h...@climate.apache.org wrote: I¹m honestly not sure what you are talking about, Not even one wee bit ;) maybe some of my replies will bridge the gap between you not being sure about what I am talking about Vs disagreeing with what I am trying to say. and to be honest, waiting 72 hours before committing everything is not a project I¹d ever want to be on. Well lets for a minute consider the flip side (or another possible side) of this coin. I would never want to be on a project where patches are merged which clearly break 50% of the tests in the entire codebase. It seems to me, rather detrimental to the project codebase as well as other community members who have confidence in a codebase for patches to be committed and merged whcih essentially break everything! Additionally, this goes against all of the good practice I've ever learned since stumbling across TheASF some 6 years ago! I can't word this opinion any other way. I'll add a +1 to not being interested on working on a project where the build is broken all the time because people can't be bothered to run the test suite. It wastes a huge amount of limited developer resource tracking down bugs for people. If you don't know how to run the tests, just ask. It's literally 1 command to run the entire test suite for the OCW package. If you're not sure just ask and people will help you out!! I don't know a single person here who wouldn't be willing to offer guidance and I know I have personally offered up my time to help out more times than I can count. Like I said - if I care about a review, or want something to be seen by someone, fine, I can choose to ask for it. Absolutely correct. I agree. It shouldn¹t be *imposed on me*. It seems like imposed is a strong word given the sentiment of the thread and the openness of Mike to open it up initially to how people want
Re: Project Workflow
When talking about project workflow, I think it makes sense to look at it in terms of value. Those of us with a technical background see value in a stable code base, consistent architecture, predictable release cycle, etc. Those of us with a scientific background see value in scientific soundness, applicability to research problems, ease of use, etc. We're both right, of course: all of it contributes to the credibility of the project. As a PMC we are responsible for inspiring our user base with our code - and our community - to share their insight on our mailing list and/or contribute their efforts to our code base. Where process helps, it should be adopted; where process hinders this, it should be reconsidered. Contributions of scientifically defensible code are really important at this stage and we should do everything possible to encourage them because it will build the community. Bugs will be found, tests can be written that duplicate them, patches can be produced that fix them, and the process will organize and gather momentum along the way. We grant committer rights to folks on OCW because we trust them to be good stewards of the code, to act in the best interest of the project, and to be a conduit for shepherding contributions from the broader community along. We have mentors to help committers who are new to the Apache Way. Where there's confusion, a mentor’s response should be individualized, geared towards growing that committer's individual skills. I think this is vastly preferable to a process that introduces impediments, however trivial, to free-flowing contribution. Finally, those who do, decide. I haven’t done much lately, so take this for whatever it’s worth. I see so much value in OCW, and respect the folks who have dedicated their time to getting things to where they are today. The passion in this thread reminds me that we’ve got a strong PMC and can certainly work this out. Best, Andrew. On 5/13/15 5:12 AM, Cameron Goodale wrote: Good Morning Kyo, I am sorry for your frustration in this matter, and I appreciate you sticking it out with us. I'll let the git professionals answer the code related stuff and I'll handle the acronyms. CTR - Commit then review ( doing code review after it has been committed) RTC - Review then Commit ( waiting for a review of the code, then committing it to the repo) PMC - Project Management Committee (http://www.apache.org/dev/pmc.html) I admit it took me about 2 or 3 emails before CTR and RTC became clear to me. Best Regards, Cam On Wed, May 13, 2015 at 3:54 AM, Lee, Kyo (329C-Caltech) huikyo@jpl.nasa.gov wrote: Dear all, Finally I read all of this thread. Now I am writing to express my sincere apologies for my words and failure to control my anger. I realized that some of my behavior inappropriate and disrespectful. I hope that you will allow me the opportunity to express my apology again in person this Thursday. I will appreciate if someone can answer the following two questions. - How can I commit to the Git repo at the ASF directly? Is this what I should do? What makes this push different from Github pull request? $ cd climate $ git remote add apache https://git-wip-us.apache.org/repos/asf/climate.git $ git push apache $branch - What is canonical source code? There are also some silly questions. What do RTC, CTR and PMC stand for? Without these abbreviations, it would be much easier to understand some of email. Your sincerely, Kyo On 5/12/15, 10:36 PM, Cameron Goodale good...@apache.org wrote: Wow! There sure is a lot of passion and fire in this thread, so let me put on my asbestos small clothes and wade into the pool. Dear Kyo, When you say that this discussion is a waste of your time it makes me sad. Sure sitting here the last 20 minutes reading this thread could be seen as a waste of my time too, but I did it and I am writing this because I care about my friends on this project, including you. I really hope you were writing out of frustration earlier, and see that this isn't a waste of your time. To everyone else that reads this, My sincere hope is that everyone on this project wants what is best for the project, best community, best climate tools, best code, best version control, etc... so when tempers flare and people are using ALL CAPS, please try to slow it down a notch and try to see it from the other person's perspective. Thanks for reading, Cameron On Tue, May 12, 2015 at 9:57 PM, Michael Joyce jo...@apache.org wrote: Thank you for the great responses Lewis. You put into words what I haven't seemed to be able to type out today. Few comments below (it's really mostly just me typing +100 to be honest though) -- Jimmy On Tue, May 12, 2015 at 8:29 PM, Lewis John Mcgibbney lewis.mcgibb...@gmail.com wrote: Hi Chris, Please see replies inline On Tue, May 12, 2015 at 6:10 PM, dev-digest-h...@climate.apache.org wrote: I¹m honestly not sure what you are talking
Re: Project Workflow
Hi Mike, Yes, I won¹t hesitate to ask your help. Thank you as always. Hi Cam and everyone, I would like to clarify that the time wasting discussion is not about the project workflow at all. I am sorry, but I have been too busy to read any email thread on the workflow. The discussion in my email is about the github pull requests. https://github.com/apache/climate/pull/196 https://github.com/apache/climate/pull/197 I wish I do not have to stay up until 2 AM just to explain simple seasonal mean calculation repeatedly. Kyo On 5/12/15, 10:54 PM, Michael Joyce jo...@apache.org wrote: Beyond my comments below I'm stepping off this thread. It's become too toxic for my liking and most of the discussion on here is helping nothing. I think getting feedback on our current workflow would be great. If you have ideas anyone please do respond (or maybe just make a new thread) so we can talk about stuff. It's important that we make it work for us! @Kyo, I've said it a million times, but it's important to say it again. If you need help with anything please don't hesitate to reach out. I'm always happy to help out when I can. You always have this list, you know where my office is, you know what my phone number is, and you have me on IM. If you're ever unsure about something or just need an extra pair of eyes to find a bug please ask. -- Jimmy On Tue, May 12, 2015 at 9:53 PM, Mattmann, Chris A (3980) chris.a.mattm...@jpl.nasa.gov wrote: Lewis, -Original Message- From: Lewis John Mcgibbney lewis.mcgibb...@gmail.com Reply-To: dev@climate.apache.org dev@climate.apache.org Date: Tuesday, May 12, 2015 at 8:29 PM To: dev@climate.apache.org dev@climate.apache.org Subject: Re: Project Workflow Hi Chris, Please see replies inline On Tue, May 12, 2015 at 6:10 PM, dev-digest-h...@climate.apache.org wrote: I¹m honestly not sure what you are talking about, Not even one wee bit ;) maybe some of my replies will bridge the gap between you not being sure about what I am talking about Vs disagreeing with what I am trying to say. Yes me saying I don¹t know what you are talking about is my way of saying I disagree with you. That clear? and to be honest, waiting 72 hours before committing everything is not a project I¹d ever want to be on. Well lets for a minute consider the flip side (or another possible side) of this coin. I would never want to be on a project where patches are merged which clearly break 50% of the tests in the entire codebase. Please, provide me a citation for this? Which PR has done that? Be specific too. Don¹t say ³the PMC². Use names, and PR #s. Not to speak for Lewis but I assume https://github.com/apache/climate/pull/195 Although that PR seems to perfectly represent receiving good feedback and making quick changes to issues that broke the build and getting everything resolved. Seemed like great collaboration to me. It seems to me, rather detrimental to the project codebase as well as other community members who have confidence in a codebase for patches to be committed and merged whcih essentially break everything! Additionally, this goes against all of the good practice I've ever learned since stumbling across TheASF some 6 years ago! I can't word this opinion any other way. Who in the world is suggesting we merge things that break 50% of the code? I¹m suggesting *we* actually merge things. Instead of having one person merge things. And mostly have others talk about how they want to merge things and then have one person end up merging things. After lots of discussion. Saying it again for the billionth time. With regards to helping merge stuff, please do. [..snip..] It shouldn¹t be *imposed on me*. It seems like imposed is a strong word given the sentiment of the thread and the openness of Mike to open it up initially to how people want to do things. I think what we are trying to determine here is whether people feel like things are being imposed upon them. If that ends up being one of the outcomes of this thread then we need to accept, address, change, implement and move on. I¹m raising it, b/c honestly, this PMC has failed to educate at least one of its members on his rights. And I am telling you that as the Champion of this project and the person who even suggested bringing it here, it¹s a collective issue. I did my part to try and educate that person today at the coffee cart, by telling Kyo this at JPL - I also noted this on the recent PR #: http://s.apache.org/to1 Kyo was on the original team when the project went into incubator correct? Seems like that certainly should have been brought up at some point. Don't know if it's fair to say that the PMC failed to educate someone who has been on the PMC since it started. I wouldn't think to check that with people honestly. Perhaps we should make sure the current list of people are aware of what's going on if this is an ongoing concern
Re: Project Workflow
I don’t think we should dictate everything be code reviewed. I’ve seen this directly lead to conversations that are relevant to development being buried in Github. Take for example your and Whitehall’s conversation(s) with Kyo that I doubt anyone here has ever seen since they aren’t even commenting on the Github. Yes, Github emails are sent to the dev list, my guess is that people ignore them. On the code review issue - Kyo (or others) shouldn’t have to endlessly debate or discuss the advantages or disadvantages of this or that before simply pushing code, and pushing tests. My general rule of thumb is that there are CTR and RTC workflows and use cases for both. RTC works great when it’s possibly controversial or when you really want someone’s eyes on your code for review. However it’s also overhead that I do not believe is needed if a developer wants to push forward and scratch his or her itch, in an area of the codebase that they are working on. The codebase is factored out enough reasonably well so that people can work on things in parallel and independently. When in doubt, ask. I’m also pretty worried since anyone that looks at the Git and project history over the last year can easily see that Mike has pretty much been doing the bulk load of the pushing and code committing here. Kim’s stepped up recently as has Kyo, which is 3 people, which is great, but I’m worried about a project with a small number of active developers (really 1 major) imposing RTC - I don’t have time to look up citations but you are free to scope these out over the ASF archives. RTC on smaller projects just leads to barriers. We need to be flexible and make it inviting for at the very least, our own developers to contribute to the project, let along attracting new people. Ross was elected in December 2014, which is great, but we need to do better. Cheers, Chris ++ Chris Mattmann, Ph.D. Chief Architect Instrument Software and Science Data Systems Section (398) NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 168-519, Mailstop: 168-527 Email: chris.a.mattm...@nasa.gov WWW: http://sunset.usc.edu/~mattmann/ ++ Adjunct Associate Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++ -Original Message- From: Michael Joyce jo...@apache.org Reply-To: dev@climate.apache.org dev@climate.apache.org Date: Tuesday, May 12, 2015 at 8:55 AM To: dev@climate.apache.org dev@climate.apache.org Subject: Project Workflow Hi folks, Since this has been brought up a few times on various tickets I thought now would be a good time to go over our project workflow and make sure it's working for us. A general overview of the workflow that we use is available at [1]. A brief overview is that: - ALL changes are pushed up to Github for code review before being merged. - If no issues are raised within a reasonable amount of time (usually 72 hours is what I stick with) those changes can be merged. In general, I've been quite happy with this workflow. We have a low enough throughput that this isn't overwhelming and I think it's great that we can get together and review each other's code. I know I appreciate the opportunity for people to find issues with my code before we merge it. I think it would be beneficial to flesh out the docs a bit more on the workflow (for instance, how to run tests should be included in there, how long to wait for a merge, etc.). So community, what do we think of our workflow? Do we like it so far? Is it working for us? Are there pain points? What don't we like? Etc. [1] https://cwiki.apache.org/confluence/display/CLIMATE/Developer+Getting+Star ted+Guide -- Jimmy
Re: Project Workflow
I believe the current workflow has inhibited progress and caused tension. Although I've been more of an observer than contributor it seems to me that when your community is small that the emphasis should be on allowing as Chris states for people to scratch their own itch. Additionally, I've seen at times that reviews have focused on minor items and that discussions often took longer than either side reaching out to the other to help get the patch in and build community and camaraderie amongst all those that are already committers. This has gone on to the extent that appears detrimental to the project. I would actively support CTR at this point so that energy and progress can be infused. Sure this could cause some technical debt to build up but community over code would seemingly once again come to the forefront which appears to be lacking at the moment. --Paul Paul Ramirez, M.S. Technical Group Supervisor Computer Science for Data Intensive Applications (398M) Instrument Software and Science Data Systems Section (398) NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 158-264, Mailstop: 158-242 Email: paul.m.rami...@jpl.nasa.govmailto:paul.m.rami...@jpl.nasa.gov Office: 818-354-1015 Cell: 818-395-8194 On May 12, 2015, at 9:33 AM, Mattmann, Chris A (3980) chris.a.mattm...@jpl.nasa.govmailto:chris.a.mattm...@jpl.nasa.gov wrote: I don’t think we should dictate everything be code reviewed. I’ve seen this directly lead to conversations that are relevant to development being buried in Github. Take for example your and Whitehall’s conversation(s) with Kyo that I doubt anyone here has ever seen since they aren’t even commenting on the Github. Yes, Github emails are sent to the dev list, my guess is that people ignore them. On the code review issue - Kyo (or others) shouldn’t have to endlessly debate or discuss the advantages or disadvantages of this or that before simply pushing code, and pushing tests. My general rule of thumb is that there are CTR and RTC workflows and use cases for both. RTC works great when it’s possibly controversial or when you really want someone’s eyes on your code for review. However it’s also overhead that I do not believe is needed if a developer wants to push forward and scratch his or her itch, in an area of the codebase that they are working on. The codebase is factored out enough reasonably well so that people can work on things in parallel and independently. When in doubt, ask. I’m also pretty worried since anyone that looks at the Git and project history over the last year can easily see that Mike has pretty much been doing the bulk load of the pushing and code committing here. Kim’s stepped up recently as has Kyo, which is 3 people, which is great, but I’m worried about a project with a small number of active developers (really 1 major) imposing RTC - I don’t have time to look up citations but you are free to scope these out over the ASF archives. RTC on smaller projects just leads to barriers. We need to be flexible and make it inviting for at the very least, our own developers to contribute to the project, let along attracting new people. Ross was elected in December 2014, which is great, but we need to do better. Cheers, Chris ++ Chris Mattmann, Ph.D. Chief Architect Instrument Software and Science Data Systems Section (398) NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 168-519, Mailstop: 168-527 Email: chris.a.mattm...@nasa.govmailto:chris.a.mattm...@nasa.gov WWW: http://sunset.usc.edu/~mattmann/ ++ Adjunct Associate Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++ -Original Message- From: Michael Joyce jo...@apache.orgmailto:jo...@apache.org Reply-To: dev@climate.apache.orgmailto:dev@climate.apache.org dev@climate.apache.orgmailto:dev@climate.apache.org Date: Tuesday, May 12, 2015 at 8:55 AM To: dev@climate.apache.orgmailto:dev@climate.apache.org dev@climate.apache.orgmailto:dev@climate.apache.org Subject: Project Workflow Hi folks, Since this has been brought up a few times on various tickets I thought now would be a good time to go over our project workflow and make sure it's working for us. A general overview of the workflow that we use is available at [1]. A brief overview is that: - ALL changes are pushed up to Github for code review before being merged. - If no issues are raised within a reasonable amount of time (usually 72 hours is what I stick with) those changes can be merged. In general, I've been quite happy with this workflow. We have a low enough throughput that this isn't overwhelming and I think it's great that we can get together and review each
Re: Project Workflow
Yeah Kyo I was trying to remove the need to debate and to move forward with code where we can. Sent from my iPhone On May 12, 2015, at 12:37 PM, Lee, Kyo (329C-Caltech) huikyo@jpl.nasa.gov wrote: Dear Chris, This debate just wastes my time. I am not simply talking about the advantages or disadvantages. The old one has serious problems. If anyones know the problem, that person would never use the module. The module of the debate is too simple to spend my time to discuss. Anyone at JPL can write the code with same functionality within an hour. Thanks, Kyo On 5/12/15, 9:33 AM, Mattmann, Chris A (3980) chris.a.mattm...@jpl.nasa.gov wrote: I don¹t think we should dictate everything be code reviewed. I¹ve seen this directly lead to conversations that are relevant to development being buried in Github. Take for example your and Whitehall¹s conversation(s) with Kyo that I doubt anyone here has ever seen since they aren¹t even commenting on the Github. Yes, Github emails are sent to the dev list, my guess is that people ignore them. On the code review issue - Kyo (or others) shouldn¹t have to endlessly debate or discuss the advantages or disadvantages of this or that before simply pushing code, and pushing tests. My general rule of thumb is that there are CTR and RTC workflows and use cases for both. RTC works great when it¹s possibly controversial or when you really want someone¹s eyes on your code for review. However it¹s also overhead that I do not believe is needed if a developer wants to push forward and scratch his or her itch, in an area of the codebase that they are working on. The codebase is factored out enough reasonably well so that people can work on things in parallel and independently. When in doubt, ask. I¹m also pretty worried since anyone that looks at the Git and project history over the last year can easily see that Mike has pretty much been doing the bulk load of the pushing and code committing here. Kim¹s stepped up recently as has Kyo, which is 3 people, which is great, but I¹m worried about a project with a small number of active developers (really 1 major) imposing RTC - I don¹t have time to look up citations but you are free to scope these out over the ASF archives. RTC on smaller projects just leads to barriers. We need to be flexible and make it inviting for at the very least, our own developers to contribute to the project, let along attracting new people. Ross was elected in December 2014, which is great, but we need to do better. Cheers, Chris ++ Chris Mattmann, Ph.D. Chief Architect Instrument Software and Science Data Systems Section (398) NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 168-519, Mailstop: 168-527 Email: chris.a.mattm...@nasa.gov WWW: http://sunset.usc.edu/~mattmann/ ++ Adjunct Associate Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++ -Original Message- From: Michael Joyce jo...@apache.org Reply-To: dev@climate.apache.org dev@climate.apache.org Date: Tuesday, May 12, 2015 at 8:55 AM To: dev@climate.apache.org dev@climate.apache.org Subject: Project Workflow Hi folks, Since this has been brought up a few times on various tickets I thought now would be a good time to go over our project workflow and make sure it's working for us. A general overview of the workflow that we use is available at [1]. A brief overview is that: - ALL changes are pushed up to Github for code review before being merged. - If no issues are raised within a reasonable amount of time (usually 72 hours is what I stick with) those changes can be merged. In general, I've been quite happy with this workflow. We have a low enough throughput that this isn't overwhelming and I think it's great that we can get together and review each other's code. I know I appreciate the opportunity for people to find issues with my code before we merge it. I think it would be beneficial to flesh out the docs a bit more on the workflow (for instance, how to run tests should be included in there, how long to wait for a merge, etc.). So community, what do we think of our workflow? Do we like it so far? Is it working for us? Are there pain points? What don't we like? Etc. [1] https://cwiki.apache.org/confluence/display/CLIMATE/Developer+Getting+Sta r ted+Guide -- Jimmy
Re: Project Workflow
in branches and committing all you want creates nightmares when the PMC who is responsible for managing the code tries to collectively work on a codebase *together* and to have shared stewardship of the code base. I don’t see that right now. I see a very small amount of people doing things every now and then, with most of the commentary on whether it’s right or wrong coming from you. And people like Kyo not even knowing if they have write access to the repository at Apache or not. That’s not an Apache project’s model and it needs to be fixed here in OCW. There are no BDFL’s in this project. We are all members of the PMC and have shared rights and stewardship to the code. Those are my thoughts on this. Cheers, Chris ++ Chris Mattmann, Ph.D. Chief Architect Instrument Software and Science Data Systems Section (398) NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 168-519, Mailstop: 168-527 Email: chris.a.mattm...@nasa.gov WWW: http://sunset.usc.edu/~mattmann/ ++ Adjunct Associate Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++ -Original Message- From: Michael Joyce jo...@apache.org Reply-To: dev@climate.apache.org dev@climate.apache.org Date: Tuesday, May 12, 2015 at 11:49 AM To: dev@climate.apache.org dev@climate.apache.org Subject: Re: Project Workflow Pre-email note - 'you' is used here to collectively refer to a nonexistent person, not a specific person in this chain of emails. --- I would certainly agree that RTC COULD cause problems if it wasn't applied fairly. But it's applied generally across all commits from all contributors here. If no one gives any feedback on a patch after a while (again, I usually stick with the fairly standard 72 hours idea), merge the dang thing. That's always been my stance on it. I think applying this to all contributions regardless of status is actually more fair than most project's approaches where contributors have to submit a patch and wait for someone to hopefully come along and give a crap. Also, I think it's a huge misnomer that CTR turns into anything other than Commit then Commit some more. I have never seen someone do a code review after pushing commits on any project I've worked on. People don't want to do reviews, they're certainly not going to do it after the fact. Do early reviews always fix the problem of bugs being introduced? Of course not. Do they make it worse? Absolutely not. We've seen that early reviews prevent broken tests and bugs from being committed to our code base multiple times (and that has been on many people' contributions, including my own many times). Even if this project decided to switch to a CTR approach every one of my contributions would go through a PR. It keeps you honest with regards to running the tests and it gives people opportunities to help you make your code better. That's always a plus to me. I wish more people would actually give feed back on my contributions/PRs. That would make me really happy. Also, no committer/PMC member is excluded from merging pull requests. Everyone who is a committer/PMC can merge pull requests. So if you want to be responsible for validating that the pull request doesn't break stuff and getting it merged, please do! The jenkins build goes a long way towards helping with that and does most of the heavy lifting anyway. If anyone is worried that only Kim and I have been merging PRs then step up and help get PRs merged. Regarding conversations being buried on Github: All conversations are mapped back to the mailing list under the appropriate PR title and to the appropriate JIRA ticket (assuming the workflow laid out on the wiki is followed). I don't see the conversation being on a PR being much different than the conversation being on JIRA when someone submits a patch (a rather common workflow on other projects in my experience). I'm open to hearing how that is problematic though. My thought, if people are going to ignore github chatter, then they're probably going to ignore JIRA chatter, and they're probably not going to notice emails either. Also, we're using git. If you want to scratch your own itch, make a branch and do work. Commit locally all you want. You have a full version of the repository. It's exactly the same thing on github/asf servers. Keeping your branch up to date with master is trivially easy. When you want to push that contribution out make a pull request. It's extremely easy to branch a billion times and merge and commit locally and do your own thing. I'm not terribly certain how having a pull request/code review centric workflow hinders this?? One more final note. Apache Spark, one of the most active ASF projects uses a more complicated Github based PR-centric workflow that uses RTC. It certainly hasn't
Re: Project Workflow
not the case, and accept it - it will earn more people around this project in 10 years including yourself. Should someone every come in and be funded full time to work on this project or 10 people come to be funded full time to work on this project, if we followed only a strictly RTC approach, we would end up with what Hadoop ended up with, and the potential for something like Spark to end up with. I¹m glad you brought Spark up. I convinced them to come to Apache. I know plenty about their community that¹s great - but also some things that aren¹t. Dirty laundry that¹s emerged in public about barriers to contributions and higher bars for committers and PMC which are now split in that project. It¹s certainly a model for doing great things, but realize too - there are 124+ contributors; large VC-backed companies whose only goal is to work on software, not necessarily do science with little funding and contributions, etc., so it¹s a largely different ballgame. When you and I and Zimmy made the Bot for OCW Climate, it was with little pieces of our time - imagine if I had you spend your time writing up the ENTIRE plan (which I never actually saw a document describing mind you) for the OCW bot before I asked you to look at the Spark one and do it? Would you have had an easy time writing up the plan FIRST before doing the work? 3. To the point about JIRA and Github notifications being as easy to spot as plain old email that simply is not backed up by fact. In fact look at Spark - they had to create a completely separate mailing list to handle their conversations because honestly dev@s.a.o was impossible to be subscribed on due to all the automated nonsense. I am still unsubscribed to dev@s.a.o to this day b/c of that. In addition, look up emails over the 15+ year history of this foundation (you have access now as a member) - JIRA and SVN auto commits and Git auto notifications and conversations have proven NOT to be very easy to follow as something with a simple subject line, written by a human being and not a bot. 4. RE: Git - committing locally is fine, but the ASF only cares about code on the ASF¹s hardware. Github (in the guide you referenced) is NOT the canonical source for Apache OCW. The canonical source is the writeable Git repos at the ASF. This was one of the major concerns about Git that the ASF had in initially implementing it. Storing stuff in branches and committing all you want creates nightmares when the PMC who is responsible for managing the code tries to collectively work on a codebase *together* and to have shared stewardship of the code base. I don¹t see that right now. I see a very small amount of people doing things every now and then, with most of the commentary on whether it¹s right or wrong coming from you. And people like Kyo not even knowing if they have write access to the repository at Apache or not. That¹s not an Apache project¹s model and it needs to be fixed here in OCW. There are no BDFL¹s in this project. We are all members of the PMC and have shared rights and stewardship to the code. Those are my thoughts on this. Cheers, Chris ++ Chris Mattmann, Ph.D. Chief Architect Instrument Software and Science Data Systems Section (398) NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 168-519, Mailstop: 168-527 Email: chris.a.mattm...@nasa.gov WWW: http://sunset.usc.edu/~mattmann/ ++ Adjunct Associate Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++ -Original Message- From: Michael Joyce jo...@apache.org Reply-To: dev@climate.apache.org dev@climate.apache.org Date: Tuesday, May 12, 2015 at 11:49 AM To: dev@climate.apache.org dev@climate.apache.org Subject: Re: Project Workflow Pre-email note - 'you' is used here to collectively refer to a nonexistent person, not a specific person in this chain of emails. --- I would certainly agree that RTC COULD cause problems if it wasn't applied fairly. But it's applied generally across all commits from all contributors here. If no one gives any feedback on a patch after a while (again, I usually stick with the fairly standard 72 hours idea), merge the dang thing. That's always been my stance on it. I think applying this to all contributions regardless of status is actually more fair than most project's approaches where contributors have to submit a patch and wait for someone to hopefully come along and give a crap. Also, I think it's a huge misnomer that CTR turns into anything other than Commit then Commit some more. I have never seen someone do a code review after pushing commits on any project I've worked on. People don't want to do reviews, they're certainly not going to do it after the fact. Do early reviews always fix the problem of bugs being introduced? Of course
Re: Project Workflow
Hi Kyo, On Tue, May 12, 2015 at 3:05 PM, dev-digest-h...@climate.apache.org wrote: This debate just wastes my time. ! Why ! I am not simply talking about the advantages or disadvantages. Advantages or disadvantages of what? The Workflow? The old one has serious problems. The old what? The Workflow? It is if the workflow you're referring to then I think it is the current workflow as I am not sure anything has changed as of yet. I understand that PMC is discussing how we can meet consensus over a workflow which works well. If anyones know the problem, that person would never use the module. What module? What problem? Can you be more verbose? I understand that you see this as a waste of your time, but for the rest of us I can say with certainty that it is not a waste of time. It is being treated with importance. The module of the debate is too simple to spend my time to discuss. Which module is this? You don't really need to discuss it. However if you feel like doing so then maybe a pull request would be better that way we have documentation for it if it does not already exist? This is up to you of course. Anyone at JPL can write the code with same functionality within an hour. Please remember that there are more people on this mailing list and using OCW than at JPL. In fact there are 29 Committers with write access to the codebase and 42 subscribers to this list in all. There is actually an increase in dev@ subscribers since the last reporting period. 42 subscribers (up 1 in the last 3 months). It may be helpful for the reasonable community of subscribers that we have to detail more thoroughly what the core problems are. There may be others experiencing these problems as well. Thanks in advance Kyo for your patience and understanding. Lewis
Re: Project Workflow
Beyond my comments below I'm stepping off this thread. It's become too toxic for my liking and most of the discussion on here is helping nothing. I think getting feedback on our current workflow would be great. If you have ideas anyone please do respond (or maybe just make a new thread) so we can talk about stuff. It's important that we make it work for us! @Kyo, I've said it a million times, but it's important to say it again. If you need help with anything please don't hesitate to reach out. I'm always happy to help out when I can. You always have this list, you know where my office is, you know what my phone number is, and you have me on IM. If you're ever unsure about something or just need an extra pair of eyes to find a bug please ask. -- Jimmy On Tue, May 12, 2015 at 9:53 PM, Mattmann, Chris A (3980) chris.a.mattm...@jpl.nasa.gov wrote: Lewis, -Original Message- From: Lewis John Mcgibbney lewis.mcgibb...@gmail.com Reply-To: dev@climate.apache.org dev@climate.apache.org Date: Tuesday, May 12, 2015 at 8:29 PM To: dev@climate.apache.org dev@climate.apache.org Subject: Re: Project Workflow Hi Chris, Please see replies inline On Tue, May 12, 2015 at 6:10 PM, dev-digest-h...@climate.apache.org wrote: I’m honestly not sure what you are talking about, Not even one wee bit ;) maybe some of my replies will bridge the gap between you not being sure about what I am talking about Vs disagreeing with what I am trying to say. Yes me saying I don’t know what you are talking about is my way of saying I disagree with you. That clear? and to be honest, waiting 72 hours before committing everything is not a project I’d ever want to be on. Well lets for a minute consider the flip side (or another possible side) of this coin. I would never want to be on a project where patches are merged which clearly break 50% of the tests in the entire codebase. Please, provide me a citation for this? Which PR has done that? Be specific too. Don’t say “the PMC”. Use names, and PR #s. Not to speak for Lewis but I assume https://github.com/apache/climate/pull/195 Although that PR seems to perfectly represent receiving good feedback and making quick changes to issues that broke the build and getting everything resolved. Seemed like great collaboration to me. It seems to me, rather detrimental to the project codebase as well as other community members who have confidence in a codebase for patches to be committed and merged whcih essentially break everything! Additionally, this goes against all of the good practice I've ever learned since stumbling across TheASF some 6 years ago! I can't word this opinion any other way. Who in the world is suggesting we merge things that break 50% of the code? I’m suggesting *we* actually merge things. Instead of having one person merge things. And mostly have others talk about how they want to merge things and then have one person end up merging things. After lots of discussion. Saying it again for the billionth time. With regards to helping merge stuff, please do. [..snip..] It shouldn’t be *imposed on me*. It seems like imposed is a strong word given the sentiment of the thread and the openness of Mike to open it up initially to how people want to do things. I think what we are trying to determine here is whether people feel like things are being imposed upon them. If that ends up being one of the outcomes of this thread then we need to accept, address, change, implement and move on. I’m raising it, b/c honestly, this PMC has failed to educate at least one of its members on his rights. And I am telling you that as the Champion of this project and the person who even suggested bringing it here, it’s a collective issue. I did my part to try and educate that person today at the coffee cart, by telling Kyo this at JPL - I also noted this on the recent PR #: http://s.apache.org/to1 Kyo was on the original team when the project went into incubator correct? Seems like that certainly should have been brought up at some point. Don't know if it's fair to say that the PMC failed to educate someone who has been on the PMC since it started. I wouldn't think to check that with people honestly. Perhaps we should make sure the current list of people are aware of what's going on if this is an ongoing concern? There's certainly a decent number of people on the PMC who aren't from a software background so that might be useful. I consider a grace period as a politeness as well as a duration which people can gauge the contribution(s) and comment accordingly. That is all. It is not so people can shoot it down. That would be detrimental indeed. Realize in CTR - they can do the same thing. We are using a version control system. Since when has code committed to a version control system been something that’s difficult to revert? You can do the exact same thing in CTR. With less barrier. Want
Re: Project Workflow
Lewis,
-Original Message-
From: Lewis John Mcgibbney lewis.mcgibb...@gmail.com
Reply-To: dev@climate.apache.org dev@climate.apache.org
Date: Tuesday, May 12, 2015 at 8:29 PM
To: dev@climate.apache.org dev@climate.apache.org
Subject: Re: Project Workflow
Hi Chris,
Please see replies inline
On Tue, May 12, 2015 at 6:10 PM, dev-digest-h...@climate.apache.org
wrote:
I’m honestly not sure what you are talking about,
Not even one wee bit ;) maybe some of my replies will bridge the gap
between you not being sure about what I am talking about Vs disagreeing
with what I am trying to say.
Yes me saying I don’t know what you are talking about is my way
of saying I disagree with you. That clear?
and to be honest,
waiting 72 hours before committing everything is not a project I’d
ever want to be on.
Well lets for a minute consider the flip side (or another possible side)
of
this coin. I would never want to be on a project where patches are merged
which clearly break 50% of the tests in the entire codebase.
Please, provide me a citation for this? Which PR has done
that? Be specific too. Don’t say “the PMC”. Use names, and PR #s.
It seems to
me, rather detrimental to the project codebase as well as other community
members who have confidence in a codebase for patches to be committed and
merged whcih essentially break everything! Additionally, this goes against
all of the good practice I've ever learned since stumbling across TheASF
some 6 years ago! I can't word this opinion any other way.
Who in the world is suggesting we merge things that break 50% of the code?
I’m suggesting *we* actually merge things. Instead of having one person
merge
things. And mostly have others talk about how they want to merge things and
then have one person end up merging things. After lots of discussion.
[..snip..]
It shouldn’t be *imposed on me*.
It seems like imposed is a strong word given the sentiment of the thread
and the openness of Mike to open it up initially to how people want to do
things. I think what we are trying to determine here is whether people
feel
like things are being imposed upon them. If that ends up being one of the
outcomes of this thread then we need to accept, address, change, implement
and move on.
I’m raising it, b/c honestly, this PMC has failed to educate at least one
of its members on his rights. And I am telling you that as the Champion of
this project and the person who even suggested bringing it here, it’s a
collective
issue. I did my part to try and educate that person today at the coffee
cart,
by telling Kyo this at JPL - I also noted this on the recent PR #:
http://s.apache.org/to1
I consider a grace period as a politeness as well as a duration which
people can gauge the contribution(s) and comment accordingly. That is all.
It is not so people can shoot it down. That would be detrimental indeed.
Realize in CTR - they can do the same thing. We are using a version control
system. Since when has code committed to a version control system been
something
that’s difficult to revert? You can do the exact same thing in CTR. With
less
barrier. Want more barrier, RTC? Use it then. Then later do CTR. It’s easy.
BTW Apache projects and their
conversation need to happen at the ASF and I’m seriously concerned
that’s not happening here. There is too much reliance on Github
for this project.
I understand what you are saying here Chris. There is a lot of development
chat going on at Github. This is on an issue-by-issue bases AFAIK however,
therefore I am of the opinion that essentially this is no different from
the same conversation happening over on Jira and the same messages being
shadowed over onto dev@.
You’re conflating the issue. This isn’t a question of Github versus JIRA.
This is a question of {Github, JIRA, auto whatever} versus {regular old
dev mail}.
The reason I say that is that (with my experiences
of mentoring Apache Usergrid) communities should not be *forced* to use
Jira over Github.
I’m not suggesting that.
The same messges are shadowed to Jira and to the Mailing
Lists. This is a nuance of the communication workflow. If this is a major
issue (we've been here before haven't we ;)) then it needs to be dealt
with. This argument has a lot of precedence at the foundation and we can
dig it up if need be.
What is your suggestion then Chris? That all correspondence is moved to
Jira? That it happens on the ML? That we find a balance between the three?
Balance of course. Important development decisions and discussions should
not be in a Pull Request conversation - it should be on the dev list, not
surrounding by a message saying “Github notification from user XXXYYY”.
Small discussion around an issue on Github? Tests passed from a bot?
Sure leave that on Github. “I don’t think X is a good implementation over
Y” - that’s dev list conversation *not* surrounded and marshaled by an
automated bot.
Yes. Flat out. We don’t VOTE 72 hours on every line of code
Re: Project Workflow
Thank you for the great responses Lewis. You put into words what I haven't seemed to be able to type out today. Few comments below (it's really mostly just me typing +100 to be honest though) -- Jimmy On Tue, May 12, 2015 at 8:29 PM, Lewis John Mcgibbney lewis.mcgibb...@gmail.com wrote: Hi Chris, Please see replies inline On Tue, May 12, 2015 at 6:10 PM, dev-digest-h...@climate.apache.org wrote: I’m honestly not sure what you are talking about, Not even one wee bit ;) maybe some of my replies will bridge the gap between you not being sure about what I am talking about Vs disagreeing with what I am trying to say. and to be honest, waiting 72 hours before committing everything is not a project I’d ever want to be on. Well lets for a minute consider the flip side (or another possible side) of this coin. I would never want to be on a project where patches are merged which clearly break 50% of the tests in the entire codebase. It seems to me, rather detrimental to the project codebase as well as other community members who have confidence in a codebase for patches to be committed and merged whcih essentially break everything! Additionally, this goes against all of the good practice I've ever learned since stumbling across TheASF some 6 years ago! I can't word this opinion any other way. I'll add a +1 to not being interested on working on a project where the build is broken all the time because people can't be bothered to run the test suite. It wastes a huge amount of limited developer resource tracking down bugs for people. If you don't know how to run the tests, just ask. It's literally 1 command to run the entire test suite for the OCW package. If you're not sure just ask and people will help you out!! I don't know a single person here who wouldn't be willing to offer guidance and I know I have personally offered up my time to help out more times than I can count. Like I said - if I care about a review, or want something to be seen by someone, fine, I can choose to ask for it. Absolutely correct. I agree. It shouldn’t be *imposed on me*. It seems like imposed is a strong word given the sentiment of the thread and the openness of Mike to open it up initially to how people want to do things. I think what we are trying to determine here is whether people feel like things are being imposed upon them. If that ends up being one of the outcomes of this thread then we need to accept, address, change, implement and move on. I consider a grace period as a politeness as well as a duration which people can gauge the contribution(s) and comment accordingly. That is all. It is not so people can shoot it down. That would be detrimental indeed. Imposed is way too strong of a word in my opinion. This is the workflow the project has laid out and used for the last year. I will agree we need better documentation on it (like this is the only thing). The intention of this thread is to make sure it's working appropriately for the project. If people don't like the workflow, let's talk about it. We're not forced to do something. It's our project. If we don't like it we can change it =D BTW Apache projects and their conversation need to happen at the ASF and I’m seriously concerned that’s not happening here. There is too much reliance on Github for this project. I understand what you are saying here Chris. There is a lot of development chat going on at Github. This is on an issue-by-issue bases AFAIK however, therefore I am of the opinion that essentially this is no different from the same conversation happening over on Jira and the same messages being shadowed over onto dev@. The reason I say that is that (with my experiences of mentoring Apache Usergrid) communities should not be *forced* to use Jira over Github. The same messges are shadowed to Jira and to the Mailing Lists. This is a nuance of the communication workflow. If this is a major issue (we've been here before haven't we ;)) then it needs to be dealt with. This argument has a lot of precedence at the foundation and we can dig it up if need be. What is your suggestion then Chris? That all correspondence is moved to Jira? That it happens on the ML? That we find a balance between the three? +1. Personally, I don't see how talking on github for code reviews is different than talking about a patch on JIRA. Yes. Flat out. We don’t VOTE 72 hours on every line of code, or every patch, or waiting for a grace period to commit things. There was no mention of VOTE'ing at all AFAIK. All commentary thus far has referred to a 72 window for community commentary that was all. After this 72 hours (not months) it is absolutely cool to commit away. BTW, it is also cool to commit away before those 72 hours. There is no Bylaws established by OCW to state anything any different. This combined with the pre-commit build for the project has kept builds stable on OCW for as
