Re: pyarrow inputstream snappy

2022-02-05 Thread Micah Kornfield
The Arrow compression libraries have too modes batch and streaming decompression. For Snappy streaming hasn't been implemented. I don't recall off the top of my head if Snappy supports streaming decompression generally. If it does, a contribution would be welcome. To just get the raw snappy

pyarrow inputstream snappy

2022-01-31 Thread Jialin Liu
Hi, I'm trying to read snappy file on HDFS using inputstream, but got the error: > > with fs.open_input_stream(read_path, **open_stream_args) as f: > File "pyarrow/_fs.pyx", line 627, in > pyarrow._fs.FileSystem.open_input_stream > File "pyarrow/_fs.pyx", line 557, in >