> > 1. experimentation/prototyping is more often than not faster to iterate if > it is closed. Allowing this model of development was a primary goal of the > design.
I agree there are advantages here. I think a large amount of speed comes from not having to gain consensus in the community. At the end of the day, I don't think there is any mechanism here to ensure everybody works in public, but I think we can at least have wording to encourage people doing extensions to post them publicly and as part of the "reservation" mechanism post a link the repo that they are being developed in, if anyone is curious. I think this would be particularly useful if there really is an intent for a number of organizations to experiment with new footer designs (but possibly also in others). Thanks, Micah On Wed, Jun 26, 2024 at 9:33 AM Alkis Evlogimenos <alkis.evlogime...@databricks.com.invalid> wrote: > Thank you for taking a look Micah. > > On the topic of openness there are various aspects that we have considered. > 1. experimentation/prototyping is more often than not faster to iterate if > it is closed. Allowing this model of development was a primary goal of the > design. > 2. when the design is final, keeping the design closed should have some > drawbacks. Duplicating content to support old readers puts some natural > incentive to make extensions official because at that point one can drop > the fat from the files and move on. Another aspect of the design is the > choice of a single extension field-id which makes the extension space tiny. > This in turn means that it is difficult to interop with others without > breaking their extensions. Ergo the easiest path to any interop is to open > the extension. > > The above, while not enforcing work to happen in the open, strike some > balance in between. > > I am open to suggestions on how to further incentivize opening extensions. > > On Wed, Jun 26, 2024 at 6:04 PM Micah Kornfield <emkornfi...@gmail.com> > wrote: > > > Hi Alkis, > > I'm generally in favor of this, my main concern/question is trying to > > encourage work to be in the open. I don't think in the long run it is > good > > for users to always have proprietary extensions inside of Parquet. > > > > IMO, I think the next steps would be to add implementations to write out > > the footer extension points. > > > > Thanks, > > Micah > > > > On Mon, Jun 24, 2024 at 1:24 PM Alkis Evlogimenos > > <alkis.evlogime...@databricks.com.invalid> wrote: > > > > > The snafus are fixed. The original should work now. > > > > > > On Sun, 23 Jun 2024, 17:58 Alkis Evlogimenos, < > > > alkis.evlogime...@databricks.com> wrote: > > > > > > > Due to some sharing snafus with automation, please request access to > > > > comment. If you are just reading I've published this here: > > > > > > > > > > https://docs.google.com/document/d/e/2PACX-1vThXkhHNozn_p1ZZWF-nCzOtoP1lKmkaV4Legq2FaRiIgwyY2XC9AmKpBtpeF8jbBB4wfjmQ6UTg03k/pub > > > > > > > > On Fri, Jun 21, 2024 at 10:29 AM Alkis Evlogimenos < > > > > alkis.evlogime...@databricks.com> wrote: > > > > > > > >> Hey folks. > > > >> > > > >> I want to move the extension PR > > > >> <https://github.com/apache/parquet-format/pull/254> forward. > > > >> Unfortunately the discussion was spread across the PR, other threads > > and > > > >> documents making it slow to progress. To avoid further > fragmentation I > > > have > > > >> put together a document > > > >> < > > > > > > https://docs.google.com/document/d/1KkoR0DjzYnLQXO-d0oRBv2k157IZU0_injqd4eV4WiI/edit > > > > > > > >> discussing the extensions mechanism in isolation. I believe the > > document > > > >> addresses all the concerns/comments from the PR and mailing list > > > >> discussions brought forward so far. > > > >> > > > >> I propose we continue the discussion in the document and once > > everything > > > >> is addressed, we finalize the PR. > > > >> > > > >> Thank you, > > > >> > > > > > > > > > >
