Thank you for taking a look Micah. On the topic of openness there are various aspects that we have considered. 1. experimentation/prototyping is more often than not faster to iterate if it is closed. Allowing this model of development was a primary goal of the design. 2. when the design is final, keeping the design closed should have some drawbacks. Duplicating content to support old readers puts some natural incentive to make extensions official because at that point one can drop the fat from the files and move on. Another aspect of the design is the choice of a single extension field-id which makes the extension space tiny. This in turn means that it is difficult to interop with others without breaking their extensions. Ergo the easiest path to any interop is to open the extension.
The above, while not enforcing work to happen in the open, strike some balance in between. I am open to suggestions on how to further incentivize opening extensions. On Wed, Jun 26, 2024 at 6:04 PM Micah Kornfield <emkornfi...@gmail.com> wrote: > Hi Alkis, > I'm generally in favor of this, my main concern/question is trying to > encourage work to be in the open. I don't think in the long run it is good > for users to always have proprietary extensions inside of Parquet. > > IMO, I think the next steps would be to add implementations to write out > the footer extension points. > > Thanks, > Micah > > On Mon, Jun 24, 2024 at 1:24 PM Alkis Evlogimenos > <alkis.evlogime...@databricks.com.invalid> wrote: > > > The snafus are fixed. The original should work now. > > > > On Sun, 23 Jun 2024, 17:58 Alkis Evlogimenos, < > > alkis.evlogime...@databricks.com> wrote: > > > > > Due to some sharing snafus with automation, please request access to > > > comment. If you are just reading I've published this here: > > > > > > https://docs.google.com/document/d/e/2PACX-1vThXkhHNozn_p1ZZWF-nCzOtoP1lKmkaV4Legq2FaRiIgwyY2XC9AmKpBtpeF8jbBB4wfjmQ6UTg03k/pub > > > > > > On Fri, Jun 21, 2024 at 10:29 AM Alkis Evlogimenos < > > > alkis.evlogime...@databricks.com> wrote: > > > > > >> Hey folks. > > >> > > >> I want to move the extension PR > > >> <https://github.com/apache/parquet-format/pull/254> forward. > > >> Unfortunately the discussion was spread across the PR, other threads > and > > >> documents making it slow to progress. To avoid further fragmentation I > > have > > >> put together a document > > >> < > > > https://docs.google.com/document/d/1KkoR0DjzYnLQXO-d0oRBv2k157IZU0_injqd4eV4WiI/edit > > > > > >> discussing the extensions mechanism in isolation. I believe the > document > > >> addresses all the concerns/comments from the PR and mailing list > > >> discussions brought forward so far. > > >> > > >> I propose we continue the discussion in the document and once > everything > > >> is addressed, we finalize the PR. > > >> > > >> Thank you, > > >> > > > > > >
