Hi Matt, I don't think we've discussed try to schema evolution formally to the spec. It would likely be something nice to have especially for simple evolution like adding columns but it is probably a long process and would need someone to drive it (create an RFC, gaining consensus and making sure two implementations have implementation). I can also see the argument that these decisions belong with individual implementations.
I haven't had a chance to review Weston's doc but I imagine the work on Datasets once implemented could cover your use case? Thanks, Micah On Monday, November 28, 2022, Matthew Scanlon <matthew.scanlon@ exosfinancial.com> wrote: > Hi Micah, > I was wondering where the arrow project stands on this issue, as it looks > like there are not many work arounds to using > pyarrow.list_(pyarrow.struct()) as many other datatypes that would "fit > the > bill" of what a list of structs achieves raises a > pyarrow.lib.ArrowNotImplementedError when calling table_to_blocks(). > > On Tue, Nov 8, 2022 at 12:53 PM Micah Kornfield <emkornfi...@gmail.com> > wrote: > > > Hi Matthew, > > Could you give some more specifics about what language/component you are > > using. In general, Arrow at a specification level doesn't deal with > schema > > evolution. Is this in regard to Datasets or a different component? > > > > Thanks, > > Micah > > > > On Mon, Nov 7, 2022 at 5:06 PM Matthew Scanlon < > > matthew.scan...@exosfinancial.com> wrote: > > > > > Good afternoon, I wanted to reach out and open a dialog about structs, > > the > > > evolution of them in schemas, and if support for such a feature is on > the > > > road map or a hard pass for the arrow team. > > > > > > Currently, it appears structs support removing a field, but will there > be > > > support for adding fields later on? Are there any recommended patterns > > for > > > supporting such a field. For example, if a field foo is a struct with > > > sub_fields A, B and then later field C gets added, the old data can not > > be > > > loaded using the new schema. > > > > > > Thank you. > > > > > > Matthew Scanlon > > > > > > -- > > > > > > > > > Broker-Dealer services offered through Exos Securities LLC, member of > > > SIPC <http://www.sipc.org/> / FINRA <http://www.finra.org/> / > > > BrokerCheck > > > <https://brokercheck.finra.org/>/ 2022 Exos, inc. For important > > > disclosures, click here > > > <https://www.exosfinancial.com/general-disclosures>. > > > > > > > > > > > > > > > > -- > > > Broker-Dealer services offered through Exos Securities LLC, member of > SIPC <http://www.sipc.org/> / FINRA <http://www.finra.org/> / > BrokerCheck > <https://brokercheck.finra.org/>/ 2022 Exos, inc. For important > disclosures, click here > <https://www.exosfinancial.com/general-disclosures>. > > > >
