A brief update here: The current implementation status page[1] is looking quite nice. I think the amount of green on that page illustrates visually that despite rumors to the contrary the state of Parquet compatibility is actually good.
Thanks to everyone who has contributed status for their implementations [2][3][4] and more! Andrew [1]: https://parquet.apache.org/docs/file-format/implementationstatus/ [2]: https://github.com/apache/parquet-site/pull/104 [3]: https://github.com/apache/parquet-site/pull/101 [4]: https://github.com/apache/parquet-site/pull/100 On Fri, Feb 7, 2025 at 8:57 AM Andrew Lamb <[email protected]> wrote: > Thank you!. Once that gets in I will find time (or really try and find > someone else to find time) to fill it out for parquet-rs > > On Thu, Feb 6, 2025 at 12:04 AM Gang Wu <[email protected]> wrote: > >> Just a heads-up that I've created a PR [1] to update the matrix >> for parquet-cpp and parquet-java. Let me know if you have any >> suggestion. Thanks! >> >> [1] https://github.com/apache/parquet-site/pull/100 >> >> >> On Tue, Feb 4, 2025 at 10:11 PM Andrew Lamb <[email protected]> >> wrote: >> >> > I wanted to highlight that there is a PR[1] to fill out this matrix for >> > cuda . Super exciting to see this starting >> > >> > [1] https://github.com/apache/parquet-site/pull/99# >> > >> > On Thu, Jul 4, 2024 at 2:57 PM Andrew Lamb <[email protected]> >> wrote: >> > >> > > Thank you! A small step perhaps but a major step forward >> > > >> > > On Thu, Jul 4, 2024 at 1:03 PM Micah Kornfield <[email protected] >> > >> > > wrote: >> > > >> > >> I just merged it. >> > >> >> > >> Cheers, >> > >> Micah >> > >> >> > >> On Thu, Jul 4, 2024 at 2:32 AM Andrew Lamb <[email protected]> >> > >> wrote: >> > >> >> > >> > The initial feature matrix PR[1], has all outstanding comments >> > addressed >> > >> > and is ready to merge from my opinion. >> > >> > >> > >> > Could one of the committers consider merging it so we can start >> > filling >> > >> the >> > >> > matrix out? >> > >> > >> > >> > Thanks, >> > >> > Andrew >> > >> > >> > >> > [1]: https://github.com/apache/parquet-site/pull/34 >> > >> > >> > >> > >> > >> > On Wed, Jun 26, 2024 at 11:22 AM Andrew Lamb < >> [email protected]> >> > >> > wrote: >> > >> > >> > >> > > I filed a ticket [1] to discuss integration testing / feature >> > >> > > compatibility testing >> > >> > > >> > >> > > [1]: https://github.com/apache/parquet-format/issues/441 >> > >> > > >> > >> > > On Wed, Jun 26, 2024 at 10:03 AM Andrew Lamb < >> > [email protected]> >> > >> > > wrote: >> > >> > > >> > >> > >> I will file a ticket for discussion shortly >> > >> > >> >> > >> > >> On Wed, Jun 26, 2024, 06:55 Julien Le Dem <[email protected]> >> > wrote: >> > >> > >> >> > >> > >>> I agree with Antoine for the separate thread/ticket for >> > integration >> > >> > >>> testing >> > >> > >>> >> > >> > >>> For the « implementation status » PR, I agree we can merge and >> > >> > iterate. I >> > >> > >>> added a couple of comments that can easily be addressed >> > afterwards. >> > >> > >>> Julien >> > >> > >>> >> > >> > >>> On Wed, Jun 26, 2024 at 10:43 Antoine Pitrou < >> [email protected]> >> > >> > wrote: >> > >> > >>> >> > >> > >>> > >> > >> > >>> > IMHO, we should either start a dedicated discussion thread >> for >> > >> > >>> > integration testing, or open a GH issue and discuss it there. >> > >> > >>> > >> > >> > >>> > Regards >> > >> > >>> > >> > >> > >>> > Antoine. >> > >> > >>> > >> > >> > >>> > >> > >> > >>> > >> > >> > >>> > On Wed, 26 Jun 2024 09:21:33 +0200 >> > >> > >>> > Alkis Evlogimenos >> > >> > >>> > <[email protected]> >> > >> > >>> > wrote: >> > >> > >>> > > It would be nice if the integration suite specifies how a >> > driver >> > >> > can >> > >> > >>> be >> > >> > >>> > > executed. Then each implementation can provide a driver and >> > the >> > >> > suite >> > >> > >>> > will >> > >> > >>> > > use that for validation. >> > >> > >>> > > >> > >> > >>> > > By specifying both reads and writes for the driver we get a >> > lot >> > >> > more >> > >> > >>> > power. >> > >> > >>> > > Given an input we can roundtrip all combination of >> > >> readers/writers >> > >> > >>> and >> > >> > >>> > > verify they can roundtrip. >> > >> > >>> > > >> > >> > >>> > > On Tue, Jun 25, 2024 at 6:42 PM Andrew Lamb < >> > >> > >>> > [email protected]> wrote: >> > >> > >>> > > >> > >> > >>> > > > FWIW I started hacking up a prototype[1] of what a >> > >> > parquet-testing >> > >> > >>> > > > integration suite might look like if anyone is interested >> > >> > >>> > > > >> > >> > >>> > > > >> > >> > >>> > > > >> > >> > >>> > > > [1]: https://github.com/apache/arrow-rs/pull/5956 >> > >> > >>> > > > >> > >> > >>> > > > On Tue, Jun 18, 2024 at 10:39 AM Alkis Evlogimenos >> > >> > >>> > > > <[email protected]> wrote: >> > >> > >>> > > > >> > >> > >>> > > > > +1. >> > >> > >>> > > > > >> > >> > >>> > > > > I would suggest you address the comments first? I went >> > >> through >> > >> > >>> the >> > >> > >>> > open >> > >> > >>> > > > > ones and most of them make sense to me (and left few >> > >> additional >> > >> > >>> > > > comments). >> > >> > >>> > > > > >> > >> > >>> > > > > On Tue, Jun 18, 2024 at 12:42 PM Andrew Lamb < >> > >> > >>> [email protected] >> > >> > >>> > > >> > >> > >>> > > > > wrote: >> > >> > >>> > > > > >> > >> > >>> > > > > > Thank you >> > >> > >>> > > > > > >> > >> > >>> > > > > > On Mon, Jun 17, 2024 at 11:40 PM Micah Kornfield < >> > >> > >>> > > > [email protected]> >> > >> > >>> > > > > > wrote: >> > >> > >>> > > > > > >> > >> > >>> > > > > > > Hi Andrew, >> > >> > >>> > > > > > > I agree with this sentiment, I asked on the PR if >> > there >> > >> > would >> > >> > >>> > be >> > >> > >>> > > > > another >> > >> > >>> > > > > > > pass and then I can merge it. >> > >> > >>> > > > > > > >> > >> > >>> > > > > > > Cheers, >> > >> > >>> > > > > > > Micah >> > >> > >>> > > > > > > >> > >> > >>> > > > > > > On Fri, Jun 14, 2024 at 3:20 AM Andrew Lamb < >> > >> > >>> > [email protected]> >> > >> > >>> > > > > > > wrote: >> > >> > >>> > > > > > > >> > >> > >>> > > > > > > > Hello Parquet Devs, >> > >> > >>> > > > > > > > >> > >> > >>> > > > > > > > I propose we merge the first (admittedly bare >> bones) >> > >> > >>> > > > "Implementation >> > >> > >>> > > > > > > > Status" page PR [1] to the website soon. I think >> > this >> > >> > page >> > >> > >>> is >> > >> > >>> > vital >> > >> > >>> > > > > to >> > >> > >>> > > > > > > the >> > >> > >>> > > > > > > > Parquet community (and to any attempt to extend >> the >> > >> > >>> format) so >> > >> > >>> > the >> > >> > >>> > > > > > sooner >> > >> > >>> > > > > > > > the better. >> > >> > >>> > > > > > > > >> > >> > >>> > > > > > > > The reason to merge the PR now is to have a base >> > from >> > >> > which >> > >> > >>> > to >> > >> > >>> > > > build. >> > >> > >>> > > > > > > That >> > >> > >>> > > > > > > > PR is already over a year old and has so many >> > >> comments it >> > >> > >>> is >> > >> > >>> > hard >> > >> > >>> > > > to >> > >> > >>> > > > > > > follow >> > >> > >>> > > > > > > > or know what the path forward is. If we insist on >> > >> sorting >> > >> > >>> all >> > >> > >>> > the >> > >> > >>> > > > > > details >> > >> > >>> > > > > > > > out before we merge it I fear it will never >> merge. >> > >> > >>> > > > > > > > >> > >> > >>> > > > > > > > Once we have a page, I think the next steps are >> to >> > >> add a >> > >> > >>> > preamble >> > >> > >>> > > > > > > > explaining what it is for and to start trying to >> > fill >> > >> out >> > >> > >>> the >> > >> > >>> > chart >> > >> > >>> > > > > for >> > >> > >>> > > > > > > an >> > >> > >>> > > > > > > > implementation (I am happy to try for >> parquet-rs). I >> > >> > >>> suspect >> > >> > >>> > during >> > >> > >>> > > > > > that >> > >> > >>> > > > > > > > process we will have to adjust some of the charts >> > >> more. >> > >> > >>> > > > > > > > >> > >> > >>> > > > > > > > Thank you for your consideration (and thank you >> for >> > >> all >> > >> > the >> > >> > >>> > > > comments >> > >> > >>> > > > > so >> > >> > >>> > > > > > > > far) >> > >> > >>> > > > > > > > >> > >> > >>> > > > > > > > Andrew >> > >> > >>> > > > > > > > >> > >> > >>> > > > > > > > [1]: >> https://github.com/apache/parquet-site/pull/34 >> > >> > >>> > > > > > > > >> > >> > >>> > > > > > > >> > >> > >>> > > > > > >> > >> > >>> > > > > >> > >> > >>> > > > >> > >> > >>> > > >> > >> > >>> > >> > >> > >>> > >> > >> > >>> > >> > >> > >>> > >> > >> > >>> >> > >> > >> >> > >> > >> > >> >> > > >> > >> >
