Beyond my comments below I'm stepping off this thread. It's become too toxic for my liking and most of the "discussion" on here is helping nothing.
I think getting feedback on our current workflow would be great. If you have ideas anyone please do respond (or maybe just make a new thread) so we can talk about stuff. It's important that we make it work for us! @Kyo, I've said it a million times, but it's important to say it again. If you need help with anything please don't hesitate to reach out. I'm always happy to help out when I can. You always have this list, you know where my office is, you know what my phone number is, and you have me on IM. If you're ever unsure about something or just need an extra pair of eyes to find a bug please ask. -- Jimmy On Tue, May 12, 2015 at 9:53 PM, Mattmann, Chris A (3980) < chris.a.mattm...@jpl.nasa.gov> wrote: > Lewis, > > -----Original Message----- > > From: Lewis John Mcgibbney <lewis.mcgibb...@gmail.com> > Reply-To: "dev@climate.apache.org" <dev@climate.apache.org> > Date: Tuesday, May 12, 2015 at 8:29 PM > To: "dev@climate.apache.org" <dev@climate.apache.org> > Subject: Re: Project Workflow > > >Hi Chris, > >Please see replies inline > > > >On Tue, May 12, 2015 at 6:10 PM, <dev-digest-h...@climate.apache.org> > >wrote: > > > >> > >> I’m honestly not sure what you are talking about, > > > > > >Not even one wee bit ;) maybe some of my replies will bridge the gap > >between you not being sure about what I am talking about Vs disagreeing > >with what I am trying to say. > > Yes me saying I don’t know what you are talking about is my way > of saying I disagree with you. That clear? > > > > > > >> and to be honest, > >> waiting 72 hours before committing everything is not a project I’d > >> ever want to be on. > > > > > >Well lets for a minute consider the flip side (or another possible side) > >of > >this coin. I would never want to be on a project where patches are merged > >which clearly break > 50% of the tests in the entire codebase. > > Please, provide me a citation for this? Which PR has done > that? Be specific too. Don’t say “the PMC”. Use names, and PR #s. > Not to speak for Lewis but I assume https://github.com/apache/climate/pull/195 Although that PR seems to perfectly represent receiving good feedback and making quick changes to issues that broke the build and getting everything resolved. Seemed like great collaboration to me. > > >It seems to > >me, rather detrimental to the project codebase as well as other community > >members who have confidence in a codebase for patches to be committed and > >merged whcih essentially break everything! Additionally, this goes against > >all of the good practice I've ever learned since stumbling across TheASF > >some 6 years ago! I can't word this opinion any other way. > > Who in the world is suggesting we merge things that break 50% of the code? > I’m suggesting *we* actually merge things. Instead of having one person > merge > things. And mostly have others talk about how they want to merge things and > then have one person end up merging things. After lots of discussion. > Saying it again for the billionth time. With regards to helping merge stuff, please do. > [..snip..] > > > > > >> It shouldn’t be *imposed on me*. > > > > > >It seems like imposed is a strong word given the sentiment of the thread > >and the openness of Mike to open it up initially to how people want to do > >things. I think what we are trying to determine here is whether people > >feel > >like things are being imposed upon them. If that ends up being one of the > >outcomes of this thread then we need to accept, address, change, implement > >and move on. > > I’m raising it, b/c honestly, this PMC has failed to educate at least one > of its members on his rights. And I am telling you that as the Champion of > this project and the person who even suggested bringing it here, it’s a > collective > issue. I did my part to try and educate that person today at the coffee > cart, > by telling Kyo this at JPL - I also noted this on the recent PR #: > > http://s.apache.org/to1 Kyo was on the original team when the project went into incubator correct? Seems like that certainly should have been brought up at some point. Don't know if it's fair to say that the PMC failed to educate someone who has been on the PMC since it started. I wouldn't think to check that with people honestly. Perhaps we should make sure the current list of people are aware of what's going on if this is an ongoing concern? There's certainly a decent number of people on the PMC who aren't from a software background so that might be useful. > > > > >I consider a grace period as a politeness as well as a duration which > >people can gauge the contribution(s) and comment accordingly. That is all. > >It is not so people can shoot it down. That would be detrimental indeed. > > Realize in CTR - they can do the same thing. We are using a version control > system. Since when has code committed to a version control system been > something > that’s difficult to revert? You can do the exact same thing in CTR. With > less > barrier. Want more barrier, RTC? Use it then. Then later do CTR. It’s easy. > > > > > > >> BTW Apache projects and their > >> conversation need to happen at the ASF and I’m seriously concerned > >> that’s not happening here. There is too much reliance on Github > >> for this project. > >> > > > >I understand what you are saying here Chris. There is a lot of development > >chat going on at Github. This is on an issue-by-issue bases AFAIK however, > >therefore I am of the opinion that essentially this is no different from > >the same conversation happening over on Jira and the same messages being > >shadowed over onto dev@. > > You’re conflating the issue. This isn’t a question of Github versus JIRA. > This is a question of {Github, JIRA, auto whatever} versus {regular old > dev mail}. > > > The reason I say that is that (with my experiences > >of mentoring Apache Usergrid) communities should not be *forced* to use > >Jira over Github. > > I’m not suggesting that. > > > The same messges are shadowed to Jira and to the Mailing > >Lists. This is a nuance of the communication workflow. If this is a major > >issue (we've been here before haven't we ;)) then it needs to be dealt > >with. This argument has a lot of precedence at the foundation and we can > >dig it up if need be. > >What is your suggestion then Chris? That all correspondence is moved to > >Jira? That it happens on the ML? That we find a balance between the three? > > Balance of course. Important development decisions and discussions should > not be in a Pull Request conversation - it should be on the dev list, not > surrounding by a message saying “Github notification from user XXXYYY”. > Small discussion around an issue on Github? Tests passed from a bot? > Sure leave that on Github. “I don’t think X is a good implementation over > Y” - that’s dev list conversation *not* surrounded and marshaled by an > automated bot. > > > > > > >> > >> Yes. Flat out. We don’t VOTE 72 hours on every line of code, or every > >> patch, > >> or waiting for a grace period to commit things. > > > > > >There was no mention of VOTE'ing at all AFAIK. All commentary thus far has > >referred to a 72 window for community commentary that was all. After this > >72 hours (not months) it is absolutely cool to commit away. BTW, it is > >also > >cool to commit away before those 72 hours. > > Please read what you just said. I have to wait 72 hours, then it’s cool. > But > before 72 hours it’s cool too. Huh? > The 72 hours is a cursory period that I said was my personal metric. Please check the OP again. There seems to be some confusion about this. > > > There is no Bylaws established > >by OCW to state anything any different. This combined with the pre-commit > >build for the project has kept builds stable on OCW for as long as I can > >remember. > > Guess who suggested making that pre-commit build and told MikeJ to look at > the way Spark did it? You’re not telling me anything I don’t already know > Lewis. > > >It seems to be doing a reasonable job at keeping the test suite > >stable and passing successfully. I would have thought that this practice > >would have been fine given the fact that comments typically come in before > >72 hours and I've seen a bunch of patches committed before 72 hours as > >well. > > Citations? Were they self patches from Mike to himself? Were they Mike and > Kim > who clearly seem to agree with each other - Kim committing Mike’s patches > or > vice versa? > > > >[..snip..] > > > > > >> If I want to add a test. > >> Update > >> something that isn’t being used in the code base, or that doesn’t even > >>have > >> tests to show how it works one way or another? Someone should be able to > >> press > >> forward on that. Releases? 72 hours. New PMC/committers? 72 hours. A new > >> JIRA ticket, > >> etc.? Not sure we need that. > >> > > > >I think there is an issue here though. It's not this type of thing that 72 > >hours is in place for. It's proposed patches which break the build and > >test > >suite and mean that others working off of master cannot keep up to date > >with master. That is what the workflow guards against. Someone please > >correct me if I am wrong. > > What is to guard against? That’s my question. What precisely are we > guarding > against? Do we not trust the version control system? What users have > complained > about a broken build thus far? Please name a user, not a developer. > > > > >[..snip..] > > > > > >> And Kyo not even knowing that > >> he has direct commit access to the Git repo at the ASF > > > > > >That's a clear failure of the PMC and Kyo's introduction to the PMC in > >communicating to Kyo that he has commit rights to the canonical source > >code > >and that he can essentially commit whenever and whatever he likes. That > >seriously needs to be addressed. > > That’s the problem - no he can’t. You can’t have it both ways. If PMC > has a 72 hour wait period then there is a barrier to him committing > whenever > he likes. If he must discuss everything there is a barrier to him > committing > whatever he likes. > > So, which is it? > Open PR, wait a reasonable amount of time for people to check stuff out, merge it. That's the workflow. "Reasonable amount of time" is at the discretion of the person getting on and deciding to merge stuff. 72 hours was my personal goto that was listed as an example in the OP. > Here’s my proposal. We start *TRUSTING* each other, and start > doing things and moving forward instead of worrying about PR discussion > and broken tests, and pointing people to read workflow wiki pages (that > you seem to be citing, but have stated you haven’t read later on in this > email) that reference Github as the canonical source. > Again. Where does it say this? Because I just read through it again and can't find what you're talking about. > > > > > >> , and honestly a > >> guide > >> that I was pointed to that says the primary source code base for the > >> project > >> is at Github (newsflash: it’s not - that’s where our *mirror* is). > >> > > > >I've never seen it so no comment. I am aware that canonical source is at > >Apache. > > Huh? You are citing the “workflow” earlier - have you read it or not? > > > > >[..snip..] > > > > > >> Lewis as a member of the foundation > >> I’m sure you’re privy to the recent strife and discussion related to > >>this > >> over > >> the years > > > > > >Absolutely I am. I've also however seen extremely successful > >Apache-compliant Github workflows dramatically increasing development and > >interest in codebases. Usergrid is an excellent example of that. We really > >struggled over there for a number of months with an entire PPMC nearly > >opting to leave the Incubator due to what they saw as ridiculous > >constraints upon what Infra wanted them to do. My justification for > >getting > >involved in this thread is because I felt I learned a lot from that > >experience and I hope I can help out here! > > This is not a Github problem. It’s a community problem, that includes > issues with Github, amongst others. Github has worked great on Nutch and > Tika - and in certain interaction types that I’m not seeing on OCW lately, > it could work well too. I’m calling out things I don’t think are working > well. > > > > > > >> - a project’s sole source of activity and conversation related to > >> development cannot be automated emails from bots. We should probably be > >> discussing dev related stuff in emails. That’s still the lowest common > >> denominator. > >> > > > >I checked out the mailing list archives and almost every message is > >generated automatically. So your point is utterly valid. I would ask if > >you > >if you think disassociating all contextual development communication from > >Jira or any other issue tracker is a wise thing to do? > > Strawman. > You: Are you citing a problem with X, so let’s do ^X. > > Ever heard of gradients that lie in-between? > > >[..snip..] > > > > > >> I see PRs from Kyo getting many suggested > >> revisions from Whitehall and Joyce - and then I see similarly some > >>issues > >> when they try and push code. I see Mike being the guy to integrate and > >>push > >> PRs after review (his own and other people’s). That’s scarily like a > >>BDFL. > >> > > > >I don't see it this way. I would be pretty convinced that Mike can speak > >up > >here and state that he would wish others to commit their own patches. > > The instructions on how to merge are in that wiki document that is constantly cited in this thread right below how to make a pull request. I haven't seen any emails about being confused so I'm not certain why people would assume people are confused? > They would if they knew how, and weren’t afraid that the PR they filed > would get 5-6 suggestions before committing a simple date/time function. > C’mon. > OCW is not Nutch. It’s not Hadoop. Let’s use Software Metrics and do a full > SLOC count and let’s put it Ohloh and see how many proposed developer hours > have been invested in it versus Nutch and/or Hadoop and/or Tika, etc. It’s > not > even close. > Also, OCW is already on Ohloh but they have some issues with their repo updates. I gave up a long time ago trying to get them to refresh their stuff. Unfortunately they weren't very responsive. All the login info was sent to private@ a long time ago I'm pretty sure. If you cant find it let me know, I'm sure it's around somewhere. > > You have to be able to adapt in projects and to be able to lower the > barrier > when necessary - I absolutely understand in Nutch having 1 or 2 iterations > sometimes on patches and on PRs and stuff. We have 5-6 interested > developers > doing things right now - and iterating and PRs and stuff and conversation > is > a great way in a distributed team across timezones and so forth to work > together. > We have nowhere near that situation with OCW right now. The active people > working on > OCW are all in the same time zone; 100% all work for the same company, and > of which there is 1 primary active developer; with 1 or 2 occasional > developers > in terms of contributions that submit things that are merged by the 1 > primary > active developer. > > > > > > > >> Yes I said it. Is Mike the merge master? > > > > > >Not at all. If you check out my other thread I sent in reply to Kyo, there > >are 29 Committers with write access to the codebase and 42 subscribers to > >this list in all. > > Red-herring. > > They are not contributing to the project right now. > > > >That is a hellish impressive number of people to have on > >the PMC. The truth though Chris is that is it were not for Mike then I > >would have had nothing committed to the codebase! > > This ignores the heritage of contributions that existed long > before Mike Joyce. > > >[..snip..] > > > > > >> He is a PMC > >> member and I’m sorry and not going to dance around the issue anymore - > >>he’s > >> not being treated like a PMC member. And I’m bringing it up and not > >> sweeping > >> it under the carpet. > >> > > > >This is a foreign concept to me and I was not aware that Kyo was not being > >treated as a PMC member. Unless I have missed this corresponence happening > >on list then I was not aware of it. > > It’s all on the Github - treating people like we trust them involves taking > a leap of faith sometimes on what they are doing. And trusting it’ll be OK. > That’s my impression reading the Github. You may disagree. > > >[..snip..] > > > > > >> He’s finding it difficult to contribute. > > > > > >Well we need to help with that. That is the only way forward. The mailing > >list is open for this kind of communication. Frustrations are all too > >often > >voiced here which is probably too late. I would like to call out Kyo > >personally and state that if you read this, and you are having an issue > >contributing to the OCW codebase over and above community comments then > >please let us know. We can either sit down one-to-one, in company or else > >the PMC can maybe even do a casual hangout and we can resole the committer > >issues. > >I would like to also point you to the following resources Kyo > >http://apache.org/dev/contributors.html#providingfeedback > >In particular it details how to submit patches and provides guidance for > >what those patches should minimally do. I quote > >" > > > >change the sourcefiles to incorporate your change or addition. Make sure > >you also provide appropriate source code documentation (like javadoc for > >java sources), and follow a project's coding conventions. > > > >check the software still compiles and runs correctly > > > >run any unit or regression tests the software may have > >" > >If there is any part of this or anything else which we can aid with then > >please let us know on the list. > > > Stop citing good software coding practices. It’s community > over code here at the ASF. I studied software architecture and > development and understand plenty about it. > > I want to reduce Kyo’s frustration about contributing to the codebase. > He has stated all the discussion before being able to make a simple commit > causes his frustration. I agree with him and think there is too much > discussion > before a commit. I’m a PMC member on the project. > > > > > > >> > >> I do. It’s too long. > > > > > >Too long for what? Is development on OCW time sensitive? > > > > > >> Please let me know the sacred vow that breaking > >> a test on OCW causes. > > > > > >Non at all. It is development code afterall and hey that is what source > >code management systems are for right? Sh*t I am all for breaking project > >builds I done it many times before or many projects. I am however also for > >taking on board a Jenkins message which tells me I've broken a number of > >tests. > > Again, please tell how I am suggesting directly to ignore Jenkins > considering > I suggested to have Mike copy the Spark one and create it. Please tell me > precisely how I am suggesting to ignore Jenkins? I’m suggesting to trust > each other. HUGE difference. > > > > > > >> We have 0 users of the project. We are our own > >> users. > > > > > >This is not true Chris I am sorry. I was only talking to someone today who > >mentioned a recently committed module is being used by foreign > >researchers. > > Where are they in the last 9 months on the list? The last year? > If they are out there, I would welcome them here. Right now I can safely > say > having been on this project for a long time and monitoring the list that > not a > single non JPL email has been sent to the list in months related to any > external > users of the software. > > > >A number of us have made efforts to grow the Climate user base and I think > >we need to be very sure that no-one is using the codebase before stating > >explicitly that no-one does. > > Please make a list of known users of the code base that aren’t using > Jinwon’s > forked version. I would seriously like to know, having been one of the > principal > people to help start this project, and also present on it internationally > for > years before it even came to the ASF. > > If you have such a list that isn’t students who aren’t on the list, I would > love to see it. Users bring desire for releases, and new features, and use > cases > and I haven’t seen any of that at all brought on to this list. There is NO > ONE > that would love more users for OCW than me. Period. I brought the project > here, > don’t suggest for a second and I take extreme offense to your attempt to > try > and educate me on the users. If you know about the users, please produce a > list > and don’t speak in generalities. > > > > > > > >> I’m going to make a scary suggestion. Push all the code! > > > > > >Really ;) > > > > > >> Do > >> things! Talk on the dev list. Figure out how not to piss > >> people like Kyo off and gain their contributions even if it means > >> breaking some tests, compromising (Kyo too), but people on all sides. > >> > > > >Chris I think we are all for doing things. I hope we are getting to a > >stage > >now where we will unearth what it is that is pissing Kyo off. > > Lewis, newsflash, I’ve unearthed it. See all my emails and commentary. See > Loikith’s email. That’s why. That’s why I’m bringing this up. > > > > We all have > >an obligation as part of the PMC to support his with any committer and > >developer resources he requires. This includes potentially guiding him > >towards development neutral lists such as community@/community-dev@, etc. > >I > >can't provide help to someone if I don't understand where the > >pain/frustration stems from. > > I do. For all the reasons stated above. > > > > > > >> > >> Sorry but PMC lead is no more special than any PMC member. > > > > > >No-one said he was AFAIK. > > direct quote from you is that Mike was the “PMC Lead”. So not sure > why you used that term - so I was following from that. > > > My point was that Mike was engaged in a primary > >development role of OCW. Whereas others were/are not. > > Bingo. Which is precisely my problem with this. One developer engaged > in the primary development role of a project at Apache is a serious > concern. > > > > > > >> The person > >> has an added responsibility of filing a board report and being the > >> eyes and the ears of the board. I haven’t seen it come across that there > >> is a concern that he’s merging everything. I have a concern. I’m > >>bringing > >> it up. > >> > > > >As you are entirely entitled to do. We are by no means at crisis point > >here. There are issues which need addressing with regards to this topic. > >I've TBH never seen a PMC with 29 members and only 1 person merging code. > > This is a precise concern point, among others that I’ve raised. > > > > > > >> This PMC will have succeeded when Kyo Lee has merged and committed his > >>own > >> code to the repo. It will succeed when Mike’s not committing everyone’s > >> PRs. > >> > > > >Sounds like a sure aim to me. It is not difficult. All Kyo needs to do is > >as follows > > > >$ cd climate > >$ git remote add apache > >https://git-wip-us.apache.org/repos/asf/climate.git > >$ git push apache $branch > > You missed the part about 5-6 conversations and comments with two other > people > that seem to agree to disagree with him. Or to suggest that he do 5-6 > things > before he commit the code. With a project with this low activity, isolated > to > a single developer, it’s a recipe for disaster. I’m wondering how you are > having > a hard time seeing that? > > > > >> > >> > > >> >I would suggest that we augment the workflow to accomodate a thirst > >>step > >> > > >> >3) Please make best efforts to at least consider other community > >>comments > >> >before merging. This way we can work collaboratively to all have a > >>better > >> >understanding of the codebase. > >> > > > > > > >Do you Chris or does anyone else have an opinion or suggestion for > >augmenting the workflow or abolishing it altogether as the above thread > >would suggest? > >I think regardsless of what we end up deciding upon, we need to have it in > >black and white on the wiki. It seems like this has become a major pain > >point for Kyo less a major concern for Chris as a champion of the project > >and ongoing mentor. > > Uhhh, yes. I’ve suggested it. > > Be flexible. Trust each other. YMMV. > > >I would like to work together to establish a resolution. Kyo, it would > >mean > >A LOT if you were part of this. This also goes for the other ~25 PMC > >members. OCW is a significant imbalance in the community with regards to > >development Vs ML correspondence and physical ML presence. An observation > >and inference on my part is that this is the result of many people > >basically not having the time and or cycles to actively discuss or develop > >OCW in line with their day-to-day operations. This is not due to an overly > >convoluted commit process. > > Disagree. It’s a combination of both. The way you can attract new > contributors > is to revisit and deal with a process that has any barriers (even 1 > barrier) > and try and have none. Some Apache projects accept patches today from a > mailing > list (HTTPD) - still to this day - and credit those contributors without > giving > them 5 or 6 elements in the “Bring me a rock game” for them to do before > committing > their patch. > > Code can be continuously adapted and updated (CTR); can be peer reviewed > before > commital (RTC); and these models can work and do work well and nicely with > one > another. > > > >The fact that Kyo is struggling is our primary > >cause for concern as loosing a valuable community member is literally a > >disaster no matter what the community is and how many individuals are > >associated with the community. > > It’s even more than that. Kyo, as a PMC member, is also indicative of the > larger set of scientists that need to use OCW and to moreover to > contribute to it. > He’s having problems doing both. > > Cheers, > Chris > >
