Hi Also copying my previous response in private.
Hi > Thanks Jack for taking the time for this doc. While the Iceberg community > and PMC so far has been one of the most collaborative, and I have > personally the utmost respect for those that laid the groundwork without > which we would not be here, the project is indeed growing faster than many > anticipated. And adding bylaw will definitely give more transparency, if > that is a concern for some users for adoption, so all in all I'm in favor > of this. > Re: emeritus, it seems a common concept in the Hadoop projects, for example > Hive: https://cwiki.apache.org/confluence/display/Hive/Bylaws. It is > sometimes used voluntarily by inactive PMC's. It seems useful in cases > where you need a majority of PMC's (in the long term many PMC's do move on > and become inactive, and emeritus would mark them out of the vote count > iiuc). However, in Hive there is no need to vote to reinstate emeritus > PMC's, a simple email to private suffices, I think that would be simpler > for this project too. > Thanks > Szehon This was in response to the discussion on emeritus, looks like Jack already took this into account in the latest proposal, so it is ok with me. I'm still for tracking emeritus status, as in the long run more PMC's naturally become inactive and it is harder to pass a majority vote. In general, I'm still in favor of the bylaws to bring clarity; it looks like we now need to drill into which ones folks have problems with, looks like 'Remove committer/PMC' may require a callout for special attention as well, in the initial email's key points. Thanks Szehon On Mon, Jun 24, 2024 at 2:24 PM Fokko Driesprong <fo...@apache.org> wrote: > Hey everyone, > > Thanks Jack for setting this up, and everyone for their feedback so far. > Sharing my exact response from the private@: > > Hey Jack, >> >> Thanks for raising this, and favor of having a bylaws where we can >> formally adopt ways of working that are specific to the Iceberg project. >> For example, we have the community guidelines >> <https://iceberg.apache.org/community/#community-guidelines> that could >> be adopted into the bylaws. >> >> Putting up my ASF-member hat >> <https://apache.org/foundation/how-it-works/#hats>. I've read through >> the first iteration of the bylaws, and I would suggest removing a couple of >> things from there: >> >> - The ASF already defines a subset of the votes >> <https://www.apache.org/foundation/voting#votes-on-code-modification> that >> are written down in the document. For example, the doc defines a release >> as >> a lazy majority, but according to the ASF docs >> <https://www.apache.org/foundation/voting#votes-on-code-modification>, it >> is a majority approval. I strongly feel that we should not diverge this. >> There are already docs on voting in new committers and PMCs >> <https://community.apache.org/newcommitter.html>. I would rather >> reference those instead of copying them. >> - I do not see the point of the Emeritus status, both for committers >> and PMC. It seems to me a lot of work to keep track of it, and according >> to >> the doc, the emeritus keeps the full privileges. >> >> Before publishing the doc on the public mailing list, I would suggest >> removing at least those two to avoid a lot of noise in the discussion. I >> would also suggest starting minimal and then adopting things one by one >> (for example, the committer and PMC criteria would be a good one). >> >> I have more comments, but I don't want to go down the rabbit hole right >> away. Thanks again for raising this, and curious what others think. >> >> Kind regards, >> Fokko >> > > As mentioned in my initial response, I think there is value in the bylaws, > but I'm a firm believer in people over process (community over code?). I'll > go over the Google-doc tomorrow morning in detail. > > Kind regards, > Fokko Driesprong > > > Op ma 24 jun 2024 om 21:20 schreef Ryan Blue <b...@apache.org>: > >> Here is my original email from the thread on the private list. It echoes >> Carl's suggestion in point 5, that we should focus on adopting bylaws that >> solve challenges that we are facing in this community, rather than adopting >> bylaws en masse or from another community with different concerns. >> >> Original email: >> >> I have no major objections to adding bylaws and codifying how the >> community operates. And most of the bylaws that are in the doc seem >> reasonable enough — if we choose to adopt them. >> >> What concerns me is adding a big list of rules and changing how the >> community operates so abruptly. >> >> That concern has two aspects. First, when rules are introduced people >> tend to focus on them and apply them mechanically; I think that will hinder >> this community, which has so far run on a high degree of trust and social >> capital. >> >> A concrete example is that while you [Jack] were out on leave, we wanted >> to make progress on the materialized view spec and I pushed back on moving >> forward before hearing from you. I think that trying to reach you was the >> right call, even if a bylaw would have allowed moving on with a majority >> vote. >> >> In short: I want us to think about what we *should* do and not just what >> we *can* do. >> >> Second, I think that rules should be introduced to solve problems and >> keep the community functioning smoothly, rather than preemptively. Many of >> these bylaws probably solved problems in the Hadoop community, but are we >> facing the same issues? >> >> I’m completely open to solving issues and making things clear by adopting >> new bylaws, but I think each addition should have a clear purpose and be >> discussed individually. An omnibus set of bylaws doesn’t seem like the >> right approach to me, particularly when it changes some of our >> well-established norms. >> >> For instance, this community has always operated primarily on consensus >> for designs. I think that’s an important part of building a standard, but >> the proposed bylaws would change this to a majority. What is the concern >> this solves? Is there a reason why a lazy majority is going to help us move >> faster? I’m skeptical that it is worth the downside, which is that part of >> the community could push past objections by calling a vote. That seems like >> an unhealthy way to evolve a spec that we all need to agree on. >> >> Jack, I really appreciate you taking the time to research the bylaws and >> I know that your intent is coming from a good place. I agree that it’s >> probably time to write some things down so that it is easier for people to >> contribute and understand how this community works. My main request is that >> we do that in an incremental way: let’s document what the community already >> does today and let’s discuss improvements like the recent updates to how we >> discuss and track design proposals. >> >> Also, one final note on process: I think that because we need to discuss >> concerns that involve the potential behavior of individuals, we should keep >> this on the private list for the time being. I’d move the discussion to dev >> when we have specific bylaws that the PMC wants to discuss with the broader >> community. >> >> Ryan >> >> On Mon, Jun 24, 2024 at 11:33 AM Carl Steinbach <c...@apache.org> wrote: >> >>> + private for PMC members who may not follow dev >>> >>> 1/ I encourage the folks who have already responded on the private@ >>> thread to replay their comments here. As I noted earlier, this discussion >>> falls outside the categories that belong on the private list. >>> >>> 2/ I think adopting a set of clearly articulated bylaws and a process >>> for amending them is a net good for the Iceberg community. >>> >>> 3/ Regarding the overlap between the proposed bylaws and other ASF >>> documents, having all of the rules in one place reduces the potential for >>> confusion and also makes it easier for outsiders to understand how the >>> project actually works. >>> >>> 4/ Regarding automatic emeritus status for committers and PMC members, >>> it's been my experience on other Apache projects that having bylaws is >>> pointless if the PMC can't achieve the necessary quorums for voting and >>> that this becomes unavoidable as the project ages and the size of the PMC >>> increases. As long as it's easy for emeritus PMC/commmitters to reinstate >>> themselves, I don't see any harm in the proposed rules for automatic >>> emeritus status. >>> >>> 5/ As someone who firmly believes that Iceberg should have project >>> bylaws, I'm concerned that it may be hard to reach a consensus on the >>> current proposal because of its length and the inclusion of several >>> provisions that address situations that have yet to be encountered by this >>> community. While it may lead to more work, we're more likely to succeed if >>> we focus first on adopting a paired-down set of bylaws that includes a >>> clearly defined process for amending them in the future. >>> >>> Thanks. >>> >>> - Carl >>> >>> On Mon, Jun 24, 2024 at 12:44 PM Jack Ye <yezhao...@gmail.com> wrote: >>> >>>> Thanks for pointing to the ASF guidelines Carl, I did not know that. I >>>> had the impression of engaging with the private list first due to responses >>>> in previous devlist discussions, but I guess I landed in the right place >>>> eventually :) >>>> >>>> > In light of the recent change of company for a few committers and PMC >>>> members >>>> Sorry that is probably my wrong use of the word "in light". I just mean >>>> this event creates a good opportunity for opening this bylaws discussion. I >>>> didn't mean at all to have bylaws specifically for speculating and >>>> restricting a few specific people. I think bylaw is something essential as >>>> the project grows larger and more participants are involved in the >>>> community. >>>> >>>> -Jack >>>> >>>> >>>> >>>> On Mon, Jun 24, 2024 at 9:37 AM Ryan Blue <b...@databricks.com.invalid> >>>> wrote: >>>> >>>>> The motivation for bylaws was this: "In light of the recent change of >>>>> company for a few committers and PMC members". >>>>> >>>>> That means that we're talking about new rules based on what a few >>>>> specific people might do. Speculation like that belongs on a private list, >>>>> just like discussing actual behavior would. While no individual was >>>>> singled >>>>> out, it's clear who these committers and PMC members are. >>>>> >>>>> On Mon, Jun 24, 2024 at 9:30 AM J G <jgreene1...@gmail.com> wrote: >>>>> >>>>>> > The reason for that is that there's a long-standing norm to discuss >>>>>> the conduct of individuals only on private lists. In this case, I think >>>>>> it >>>>>> applies even though it is discussing hypothetical conduct. And note that >>>>>> I'm one of the individuals here. >>>>>> >>>>>> Respectfully, what does this mean, Ryan? No individual was even >>>>>> singled out here. This comes off as stifling discussion, not cool... >>>>>> >>>>>> On Mon, Jun 24, 2024, 9:08 AM Jack Ye <yezhao...@gmail.com> wrote: >>>>>> >>>>>>> Sorry for the confusion Ryan, this is not mistakenly sent to >>>>>>> devlist. As we discussed, this is the thread for collecting community >>>>>>> feedback, which is essential for forming bylaws with the community. >>>>>>> >>>>>>> We have that separated discussion thread in the private list, which >>>>>>> we will continue to iterate, and the vote will eventually be carried >>>>>>> out in >>>>>>> the private list as you said, according to the ASF guideline. >>>>>>> >>>>>>> -Jack >>>>>>> >>>>>>> On Mon, Jun 24, 2024 at 8:59 AM Ryan Blue >>>>>>> <b...@databricks.com.invalid> wrote: >>>>>>> >>>>>>>> Hey everyone, I think Jack mistakenly sent this to the dev list so >>>>>>>> please let's pause discussion for now. >>>>>>>> >>>>>>>> There's a thread on the private list about this in which PMC >>>>>>>> members, including me, have asked to keep it on the private list right >>>>>>>> now. >>>>>>>> >>>>>>>> The reason for that is that there's a long-standing norm to discuss >>>>>>>> the conduct of individuals only on private lists. In this case, I >>>>>>>> think it >>>>>>>> applies even though it is discussing hypothetical conduct. And note >>>>>>>> that >>>>>>>> I'm one of the individuals here. >>>>>>>> >>>>>>>> I know that there are also things here that merit discussion (like >>>>>>>> how to get PRs moving faster), but right now they are all tied up in a >>>>>>>> bundle of proposed bylaws. I've asked on the PMC list to separate the >>>>>>>> concerns and to discuss these issues individually. We will follow up >>>>>>>> with >>>>>>>> more discussion on this list. >>>>>>>> >>>>>>>> Ryan >>>>>>>> >>>>>>>> On Mon, Jun 24, 2024 at 6:19 AM Renjie Liu <liurenjie2...@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Thanks Jack for raising this, this is quite important to keep >>>>>>>>> healthy of this community. >>>>>>>>> >>>>>>>>> I agree with Ajantha about the concerns of accumulated proposals >>>>>>>>> and prs, and maybe we should have another thread to discuss about it? >>>>>>>>> >>>>>>>>> On Mon, Jun 24, 2024 at 20:37 Robert Stupp <sn...@snazy.de> wrote: >>>>>>>>> >>>>>>>>>> Thanks Jack for the proposal. >>>>>>>>>> I’m generally +1 on this. There are a few details to clarify, but >>>>>>>>>> I suspect nothing that’s controversial. >>>>>>>>>> >>>>>>>>>> On 24. Jun 2024, at 12:45, Ajantha Bhat <ajanthab...@gmail.com> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> Thank you, Jack, for your diligent work on this. >>>>>>>>>> >>>>>>>>>> This seems essential at the moment. >>>>>>>>>> >>>>>>>>>> I would like to address a couple of additional points that need >>>>>>>>>> our attention: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> *Criteria for Committership/PMC:*We've observed an inconsistency >>>>>>>>>> in how committership is granted. Contributors to sub-projects often >>>>>>>>>> attain >>>>>>>>>> committership to the main project more readily, while some who >>>>>>>>>> contribute >>>>>>>>>> significantly to the main project remain unrecognized. Although >>>>>>>>>> defining >>>>>>>>>> explicit criteria is challenging, it might be beneficial to establish >>>>>>>>>> guidelines or metrics that highlight the impact and quality of >>>>>>>>>> contributions. This could encourage more balanced and motivated >>>>>>>>>> participation across all project areas. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> *Accumulation of Proposals and PRs:*We have several proposals >>>>>>>>>> and PRs that are currently stalled despite multiple pings. Examples >>>>>>>>>> include >>>>>>>>>> the partition stats PR and proposals like table profitability, multi >>>>>>>>>> table >>>>>>>>>> transaction, secondary indexing etc. These important contributions >>>>>>>>>> are not >>>>>>>>>> making the expected progress. It might be helpful to create bylaws or >>>>>>>>>> procedures to ensure these proposals and PRs receive the necessary >>>>>>>>>> attention and are addressed promptly. This could involve setting >>>>>>>>>> timeframes >>>>>>>>>> for reviews or establishing a prioritization process. >>>>>>>>>> >>>>>>>>>> Thoughts and feedback on these suggestions would be highly >>>>>>>>>> valuable. >>>>>>>>>> >>>>>>>>>> - Ajantha >>>>>>>>>> >>>>>>>>>> On Mon, Jun 24, 2024 at 1:09 PM Jack Ye <yezhao...@gmail.com> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Hi everyone, >>>>>>>>>>> >>>>>>>>>>> In light of the recent change of company for a few committers >>>>>>>>>>> and PMC members, I hear an increasing ask from the community to >>>>>>>>>>> define >>>>>>>>>>> proper processes in Iceberg to ensure its vendor neutral stance. >>>>>>>>>>> >>>>>>>>>>> I propose that we put up a bylaws document like other projects >>>>>>>>>>> such as Apache Hadoop and Apache ORC. I think this will put people >>>>>>>>>>> at peace >>>>>>>>>>> and remove many people's concerns about the future of the project >>>>>>>>>>> and its >>>>>>>>>>> vendor-neutral stance. >>>>>>>>>>> >>>>>>>>>>> Here is a document that I have drafted that can be used as the >>>>>>>>>>> starting point (mostly just copied from the Hadoop one): >>>>>>>>>>> https://docs.google.com/document/d/1BVHbshE2dmCH8QzkeMd9PQdJ86_slavDy1YPueqNSgI/edit >>>>>>>>>>> >>>>>>>>>>> This proposal is currently undergoing review by the PMC. At the >>>>>>>>>>> same time, it is critical to also understand the feedback from the >>>>>>>>>>> community so that we can formulate the bylaws in a way that meets >>>>>>>>>>> the >>>>>>>>>>> community's needs. >>>>>>>>>>> >>>>>>>>>>> As a guide, here are a few key points that I think is worth >>>>>>>>>>> special attention and everyone's opinion: >>>>>>>>>>> >>>>>>>>>>> 0. *Bylaws or not*: first and foremost, do we think such bylaws >>>>>>>>>>> would be helpful, or do we think this is too much? >>>>>>>>>>> >>>>>>>>>>> 1. *Emeritus status*: the bylaws introduce the concept of >>>>>>>>>>> emeritus committers and PMC members, and will mark inactive >>>>>>>>>>> committers and >>>>>>>>>>> PMC members as emeritus. It is designed purely for information >>>>>>>>>>> purposes, so >>>>>>>>>>> it is easier to know who to count for during a vote. For people >>>>>>>>>>> that have >>>>>>>>>>> questions regarding the PMC members' vendor affiliations, they can >>>>>>>>>>> use this >>>>>>>>>>> information to find out details about the proportion of active PMC >>>>>>>>>>> members >>>>>>>>>>> in each organization. The emeritus status will not remove rights of >>>>>>>>>>> the >>>>>>>>>>> specific committers or PMC members, and anyone who would like to >>>>>>>>>>> re-join >>>>>>>>>>> the community just needs to report to PMC with an email to remove >>>>>>>>>>> the >>>>>>>>>>> status. >>>>>>>>>>> >>>>>>>>>>> 2. *PMC chair rotation*: the bylaws introduce an annual >>>>>>>>>>> rotation of the PMC chair. The chair is purely an administrative >>>>>>>>>>> role that >>>>>>>>>>> does not have special power, but it is undeniable that the chair is >>>>>>>>>>> usually >>>>>>>>>>> seen as the lead of the project from a public perspective. By >>>>>>>>>>> further >>>>>>>>>>> rotating this role, it makes it clear that no one in the PMC is >>>>>>>>>>> specifically leading the project direction, and the PMC as a whole >>>>>>>>>>> manages >>>>>>>>>>> the direction of Iceberg. Given that right now at least 10 PMC >>>>>>>>>>> members are >>>>>>>>>>> fully salaried workers dedicated to the Iceberg project, the >>>>>>>>>>> overhead seems >>>>>>>>>>> acceptable to perform rotations. >>>>>>>>>>> >>>>>>>>>>> 3. *Definition of subprojects*: the bylaws divide the project >>>>>>>>>>> into subprojects, in order to properly define the responsibility of >>>>>>>>>>> a >>>>>>>>>>> release manager. This is a concept learned from projects like >>>>>>>>>>> Hadoop, and I >>>>>>>>>>> think as Iceberg is growing a lot in scope, this division will also >>>>>>>>>>> help us >>>>>>>>>>> formalize different modules of the project, and properly release >>>>>>>>>>> components >>>>>>>>>>> like the table format spec, catalog spec, etc. that can be used for >>>>>>>>>>> consumption without dependency on releasing language-specific >>>>>>>>>>> libraries. >>>>>>>>>>> >>>>>>>>>>> 4. *Voting procedure*: the bylaws describe a lot about the >>>>>>>>>>> voting procedure, and how different matters in Iceberg should be >>>>>>>>>>> voted. I >>>>>>>>>>> mostly referenced other projects when writing the related sections, >>>>>>>>>>> so it >>>>>>>>>>> would be great for us to double check it with existing decisions >>>>>>>>>>> that have >>>>>>>>>>> been made in Iceberg regarding voting, as well as related rules in >>>>>>>>>>> ASF to >>>>>>>>>>> avoid any conflict. >>>>>>>>>>> >>>>>>>>>>> Really appreciate any feedback, and look forward to discussing >>>>>>>>>>> this with everyone! >>>>>>>>>>> >>>>>>>>>>> Best, >>>>>>>>>>> Jack Ye >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Ryan Blue >>>>>>>> Databricks >>>>>>>> >>>>>>> >>>>> >>>>> -- >>>>> Ryan Blue >>>>> Databricks >>>>> >>>> >> >> -- >> Ryan Blue >> >
