Ah good to know, thanks for the clarifications Neal. Clearly I haven't been keeping up very well.
On Fri, Dec 18, 2020, 09:49 Neal Richardson <neal.p.richard...@gmail.com> wrote: > A few clarifications: Feather, in it's version 2, _is_ the Arrow IPC file > format. We've kept the Feather name as a way of referring to Arrow files. > The original Feather file format, which had differences from the Arrow IPC > format, did not support compression. The Arrow IPC format may include > compression (https://issues.apache.org/jira/browse/ARROW-300), but as > Micah > brought up on the user mailing list thread, it's only the C++ > implementation and libraries using it that have implemented yet, and the > feature is not well documented yet. > > So all Arrow libraries support Feather v2 (as it is the IPC file format), > but currently only C++ (thus Python, R, and glib/Ruby) supports Feather/IPC > files with compression. > > Neal > > On Fri, Dec 18, 2020 at 8:18 AM Brian Hulette <bhule...@apache.org> wrote: > > > Hi Andrew, > > I'm glad you got this working! The javascript library only implements the > > arrow IPC spec, it doesn't have any special handling for feather and its > > compression support. It's good to know that you can read uncompressed > > feather files, but I'd only expect it to read an IPC stream or file. This > > is what I did for the Intro to Arrow JS notebook [1], see scrabble.py > here > > [2]. Note that python script was written many versions of arrow ago, I'm > > sure there's less boilerplate required for this in pyarrow 2.0. > > > > Support for feather and compression would certainly be a welcome > > contribution > > > > [1] https://observablehq.com/@theneuralbit/introduction-to-apache-arrow > > [2] > https://gist.github.com/TheNeuralBit/64d8cc13050c9b5743281dcf66059de5 > > > > On Thu, Dec 17, 2020 at 10:10 AM Andrew Clancy <n...@achren.org> wrote: > > > > > So, I figured out the issue here - I had to remove compression from the > > > pyarrow feather.write_feather(compression='uncompressed'). Is there any > > way > > > to read a compressed feather file in arrow js? > > > See the comment under the first answer here: > > > > > > > > > https://stackoverflow.com/questions/64629670/how-to-write-a-pandas-dataframe-to-arrow-file/64648955#64648955 > > > I couldn't find anything in the arrow docs or notebooks on this - I'm > > > assuming that's related to javascript compression libraries being so > > > limited. > > > > > > On Mon, 14 Dec 2020 at 19:02, Andrew Clancy <n...@achren.org> wrote: > > > > > > > Hi, > > > > > > > > I have a simple feather file created via a pandas to_feather with a > > > > datetime64[ns] column, and cannot get timestamps in javascript > > > > apache-arrow@2.0.0 > > > > > > > > See this notebook: > > > > https://observablehq.com/@nite/apache-arrow-timestamp-investigation > > > > > > > > I'm guessing I'm missing something, has anyone got any suggestions, > or > > > > decent examples of reading a file created in pandas? I've seen in > > > examples > > > > of apache-arrow@0.3.1 where dates stored as an array of 2 ints. > > > > > > > > File was created with: > > > > > > > > import pandas as pd > > > > pd.read_parquet('sample.parquet') > > > > df.to_feather('sample-seconds.feather') > > > > > > > > Final Q: I'm assuming this is the best place for this question? Happy > > to > > > > post elsewhere if there's any other forums, or if this should be a > JIRA > > > > ticket? > > > > > > > > Thanks! > > > > Andy > > > > > > > > > >
