[
https://issues.apache.org/jira/browse/ARROW-7102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17077740#comment-17077740
]
Matthew Corley edited comment on ARROW-7102 at 4/8/20, 1:53 AM:
----------------------------------------------------------------
[~apitrou] - The Python wrapping for the C++ S3 filesystem implementation (
{{pyarrow.fs.S3FileSystem}}) seems to be missing in the latest (0.16.0) macOS
wheel. Is this expected or should I open a separate issue?
Steps to repro (in a fresh venv):
*Install*
{code:bash}
❯ pip install --no-cache-dir --index-url https://pypi.python.org/pypi pyarrow
Looking in indexes: https://pypi.python.org/pypi
Collecting pyarrow
Downloading pyarrow-0.16.0-cp36-cp36m-macosx_10_9_intel.whl (40.6 MB)
|████████████████████████████████| 40.6 MB 17.2 MB/s
Collecting numpy>=1.14
Downloading numpy-1.18.2-cp36-cp36m-macosx_10_9_x86_64.whl (15.2 MB)
|████████████████████████████████| 15.2 MB 2.7 MB/s
Collecting six>=1.0.0
Downloading six-1.14.0-py2.py3-none-any.whl (10 kB)
Installing collected packages: numpy, six, pyarrow
Successfully installed numpy-1.18.2 pyarrow-0.16.0 six-1.14.0{code}
*Import:*
{code:python}
Python 3.6.9 (default, Oct 1 2019, 14:07:47)
[GCC 4.2.1 Compatible Apple LLVM 10.0.1 (clang-1001.0.46.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from pyarrow import fs
>>> dir(fs)
['FileSelector', 'FileStats', 'FileSystem', 'FileType', 'HadoopFileSystem',
'HdfsOptions', 'LocalFileSystem', 'LocalFileSystemOptions',
'SubTreeFileSystem', '_MockFileSystem', '__builtins__', '__cached__',
'__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__',
'absolute_import']
{code}
was (Author: corleyma):
[~apitrou] - The Python wrapping for the C++ S3 filesystem implementation (
{{pyarrow.fs.S3FileSystem}}) seems to be missing in the latest (0.16.0) macOS
wheel. Is this expected or should I open a separate issue?
Steps to repro (in a fresh venv):
* Install*
{code:bash}
❯ pip install --no-cache-dir --index-url https://pypi.python.org/pypi pyarrow
Looking in indexes: https://pypi.python.org/pypi
Collecting pyarrow
Downloading pyarrow-0.16.0-cp36-cp36m-macosx_10_9_intel.whl (40.6 MB)
|████████████████████████████████| 40.6 MB 17.2 MB/s
Collecting numpy>=1.14
Downloading numpy-1.18.2-cp36-cp36m-macosx_10_9_x86_64.whl (15.2 MB)
|████████████████████████████████| 15.2 MB 2.7 MB/s
Collecting six>=1.0.0
Downloading six-1.14.0-py2.py3-none-any.whl (10 kB)
Installing collected packages: numpy, six, pyarrow
Successfully installed numpy-1.18.2 pyarrow-0.16.0 six-1.14.0{code}
*Import:*
{code:python}
Python 3.6.9 (default, Oct 1 2019, 14:07:47)
[GCC 4.2.1 Compatible Apple LLVM 10.0.1 (clang-1001.0.46.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from pyarrow import fs
>>> dir(fs)
['FileSelector', 'FileStats', 'FileSystem', 'FileType', 'HadoopFileSystem',
'HdfsOptions', 'LocalFileSystem', 'LocalFileSystemOptions',
'SubTreeFileSystem', '_MockFileSystem', '__builtins__', '__cached__',
'__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__',
'absolute_import']
{code}
> [Python] Make filesystem wrappers compatible with fsspec
> --------------------------------------------------------
>
> Key: ARROW-7102
> URL: https://issues.apache.org/jira/browse/ARROW-7102
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Python
> Reporter: Tom Augspurger
> Priority: Major
> Labels: FileSystem
>
> [fsspec|https://filesystem-spec.readthedocs.io/en/latest] defines a common
> API for a variety filesystem implementations. I'm proposing a FSSpecWrapper,
> similar to S3FSWrapper, that works with any fsspec implementation.
>
> Right now, pyarrow has a pyarrow.filesystems.S3FSWrapper, which is specific
> to s3fs.
> [https://github.com/apache/arrow/blob/21ad7ac1162eab188a1e15923fb1de5b795337ec/python/pyarrow/filesystem.py#L320].
> This implementation could be removed entirely once an FSSPecWrapper is done,
> or kept as an alias if it's part of the public API.
>
> This is realted to ARROW-3717, which requested a GCSFSWrapper for working
> with google cloud storage.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
