Perspective uses arrow across the wire but internally uses it's own formats.
Tim Paine tim.paine.nyc 908-721-1185 > On Mar 19, 2021, at 09:46, Michael Lavina <michael.lav...@factset.com> wrote: > > Hey Benjamin, > > That sounds really awesome. Thank you. > > Sorry if this was already a well known thing as I am fairly new to the Arrow > ecosystem. Is there a way to track a roadmap for Arrow 4 and be involved in > that? Is there anywhere I can read more just general information on that? > > -Michael > > From: Benjamin Kietzman <bengil...@gmail.com> > Date: Friday, March 19, 2021 at 9:14 AM > To: dev <dev@arrow.apache.org> > Subject: Re: [DISCUSS] How to encode table_pivot information state in Arrow > Hi Michael, > > We are targeting grouped aggregation for 4.0 as part of a general query > engine buildout. We also intend to bring DataFrame functionality into core > Arrow (which would probably include an analog of pandas' pivot_table), but > the query engine work is a prerequisite. > > Ben Kietzman > >> On Fri, Mar 19, 2021, 08:19 Michael Lavina <michael.lav...@factset.com> >> wrote: >> >> Hey Team, >> >> Sorry if this is answered already somewhere I tried searching emails and >> issues but couldn’t find anything. I am wondering if there is a standard >> way to encode row or column pivots in Arrow? >> >> I know Pandas does it already some way >> https://urldefense.com/v3/__https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.pivot_table.html__;!!PBKjc0U4!aRYySdE5nJFh6JBpP7YXqwFlAXpHj81USUsUKdIyHn_ryLYJxyKobsgdrfhI8e_Ejvqp$<https://urldefense.com/v3/__https:/pandas.pydata.org/pandas-docs/stable/reference/api/pandas.pivot_table.html__;!!PBKjc0U4!aRYySdE5nJFh6JBpP7YXqwFlAXpHj81USUsUKdIyHn_ryLYJxyKobsgdrfhI8e_Ejvqp$> >> and there are libraries using Arrow like Perspective that may have their >> internal solution for representation of pivots >> https://urldefense.com/v3/__https://perspective.finos.org/docs/md/view.html*row-pivots__;Iw!!PBKjc0U4!aRYySdE5nJFh6JBpP7YXqwFlAXpHj81USUsUKdIyHn_ryLYJxyKobsgdrfhI8cXbLZNA$<https://urldefense.com/v3/__https:/perspective.finos.org/docs/md/view.html*row-pivots__;Iw!!PBKjc0U4!aRYySdE5nJFh6JBpP7YXqwFlAXpHj81USUsUKdIyHn_ryLYJxyKobsgdrfhI8cXbLZNA$> >> >> I am wondering if there is already a discussion or already a best practice >> or standard for encoding this information. Or alternatively is this not >> something that should be at all associated with Arrow. >> >> -Michael >> >> P.S. If anyone on the Perspective team or anyone who might know is on this >> thread I would be interested in understanding more how Perspective, >> specifically, encodes pivot information in Arrow. >> >>
