> I think we can at least have wording to encourage people doing extensions to post them publicly and as part of the "reservation" mechanism post a link the repo that they are being developed in, if anyone is curious.
Good point. I will try to come up with something in the PR - unless you beat me to it :) On Fri, Jun 28, 2024 at 7:15 AM Micah Kornfield <emkornfi...@gmail.com> wrote: > > > > 1. experimentation/prototyping is more often than not faster to iterate > if > > it is closed. Allowing this model of development was a primary goal of > the > > design. > > > I agree there are advantages here. I think a large amount of speed comes > from not having to gain consensus in the community. > > At the end of the day, I don't think there is any mechanism here to ensure > everybody works in public, but I think we can at least have wording to > encourage people doing extensions to post them publicly and as part of the > "reservation" mechanism post a link the repo that they are being developed > in, if anyone is curious. I think this would be particularly useful if > there really is an intent for a number of organizations to experiment with > new footer designs (but possibly also in others). > > Thanks, > Micah > > > > > On Wed, Jun 26, 2024 at 9:33 AM Alkis Evlogimenos > <alkis.evlogime...@databricks.com.invalid> wrote: > > > Thank you for taking a look Micah. > > > > On the topic of openness there are various aspects that we have > considered. > > 1. experimentation/prototyping is more often than not faster to iterate > if > > it is closed. Allowing this model of development was a primary goal of > the > > design. > > 2. when the design is final, keeping the design closed should have some > > drawbacks. Duplicating content to support old readers puts some natural > > incentive to make extensions official because at that point one can drop > > the fat from the files and move on. Another aspect of the design is the > > choice of a single extension field-id which makes the extension space > tiny. > > This in turn means that it is difficult to interop with others without > > breaking their extensions. Ergo the easiest path to any interop is to > open > > the extension. > > > > The above, while not enforcing work to happen in the open, strike some > > balance in between. > > > > I am open to suggestions on how to further incentivize opening > extensions. > > > > On Wed, Jun 26, 2024 at 6:04 PM Micah Kornfield <emkornfi...@gmail.com> > > wrote: > > > > > Hi Alkis, > > > I'm generally in favor of this, my main concern/question is trying to > > > encourage work to be in the open. I don't think in the long run it is > > good > > > for users to always have proprietary extensions inside of Parquet. > > > > > > IMO, I think the next steps would be to add implementations to write > out > > > the footer extension points. > > > > > > Thanks, > > > Micah > > > > > > On Mon, Jun 24, 2024 at 1:24 PM Alkis Evlogimenos > > > <alkis.evlogime...@databricks.com.invalid> wrote: > > > > > > > The snafus are fixed. The original should work now. > > > > > > > > On Sun, 23 Jun 2024, 17:58 Alkis Evlogimenos, < > > > > alkis.evlogime...@databricks.com> wrote: > > > > > > > > > Due to some sharing snafus with automation, please request access > to > > > > > comment. If you are just reading I've published this here: > > > > > > > > > > > > > > > https://docs.google.com/document/d/e/2PACX-1vThXkhHNozn_p1ZZWF-nCzOtoP1lKmkaV4Legq2FaRiIgwyY2XC9AmKpBtpeF8jbBB4wfjmQ6UTg03k/pub > > > > > > > > > > On Fri, Jun 21, 2024 at 10:29 AM Alkis Evlogimenos < > > > > > alkis.evlogime...@databricks.com> wrote: > > > > > > > > > >> Hey folks. > > > > >> > > > > >> I want to move the extension PR > > > > >> <https://github.com/apache/parquet-format/pull/254> forward. > > > > >> Unfortunately the discussion was spread across the PR, other > threads > > > and > > > > >> documents making it slow to progress. To avoid further > > fragmentation I > > > > have > > > > >> put together a document > > > > >> < > > > > > > > > > > https://docs.google.com/document/d/1KkoR0DjzYnLQXO-d0oRBv2k157IZU0_injqd4eV4WiI/edit > > > > > > > > > >> discussing the extensions mechanism in isolation. I believe the > > > document > > > > >> addresses all the concerns/comments from the PR and mailing list > > > > >> discussions brought forward so far. > > > > >> > > > > >> I propose we continue the discussion in the document and once > > > everything > > > > >> is addressed, we finalize the PR. > > > > >> > > > > >> Thank you, > > > > >> > > > > > > > > > > > > > > >
