http://git-wip-us.apache.org/repos/asf/arrow-site/blob/4d4a3202/docs/python/_sources/filesystems.rst.txt ---------------------------------------------------------------------- diff --git a/docs/python/_sources/filesystems.rst.txt b/docs/python/_sources/filesystems.rst.txt index 61c03c5..c0530f9 100644 --- a/docs/python/_sources/filesystems.rst.txt +++ b/docs/python/_sources/filesystems.rst.txt @@ -15,8 +15,8 @@ .. specific language governing permissions and limitations .. under the License. -Filesystem Interfaces -===================== +File System Interfaces +====================== In this section, we discuss filesystem-like interfaces in PyArrow. @@ -31,12 +31,14 @@ System. You connect like so: .. code-block:: python import pyarrow as pa - hdfs = pa.HdfsClient(host, port, user=user, kerb_ticket=ticket_cache_path) + fs = pa.hdfs.connect(host, port, user=user, kerb_ticket=ticket_cache_path) + with fs.open(path, 'rb') as f: + # Do something with f -By default, ``pyarrow.HdfsClient`` uses libhdfs, a JNI-based interface to the -Java Hadoop client. This library is loaded **at runtime** (rather than at link -/ library load time, since the library may not be in your LD_LIBRARY_PATH), and -relies on some environment variables. +By default, ``pyarrow.hdfs.HadoopFileSystem`` uses libhdfs, a JNI-based +interface to the Java Hadoop client. This library is loaded **at runtime** +(rather than at link / library load time, since the library may not be in your +LD_LIBRARY_PATH), and relies on some environment variables. * ``HADOOP_HOME``: the root of your installed Hadoop distribution. Often has `lib/native/libhdfs.so`. @@ -56,5 +58,33 @@ You can also use libhdfs3, a thirdparty C++ library for HDFS from Pivotal Labs: .. code-block:: python - hdfs3 = pa.HdfsClient(host, port, user=user, kerb_ticket=ticket_cache_path, - driver='libhdfs3') + fs = pa.hdfs.connect(host, port, user=user, kerb_ticket=ticket_cache_path, + driver='libhdfs3') + +HDFS API +~~~~~~~~ + +.. currentmodule:: pyarrow + +.. autosummary:: + :toctree: generated/ + + hdfs.connect + HadoopFileSystem.cat + HadoopFileSystem.chmod + HadoopFileSystem.chown + HadoopFileSystem.delete + HadoopFileSystem.df + HadoopFileSystem.disk_usage + HadoopFileSystem.download + HadoopFileSystem.exists + HadoopFileSystem.get_capacity + HadoopFileSystem.get_space_used + HadoopFileSystem.info + HadoopFileSystem.ls + HadoopFileSystem.mkdir + HadoopFileSystem.open + HadoopFileSystem.rename + HadoopFileSystem.rm + HadoopFileSystem.upload + HdfsFile
http://git-wip-us.apache.org/repos/asf/arrow-site/blob/4d4a3202/docs/python/_sources/generated/pyarrow.ChunkedArray.rst.txt ---------------------------------------------------------------------- diff --git a/docs/python/_sources/generated/pyarrow.ChunkedArray.rst.txt b/docs/python/_sources/generated/pyarrow.ChunkedArray.rst.txt index 74ed748..4b2fcb1 100644 --- a/docs/python/_sources/generated/pyarrow.ChunkedArray.rst.txt +++ b/docs/python/_sources/generated/pyarrow.ChunkedArray.rst.txt @@ -28,5 +28,6 @@ pyarrow\.ChunkedArray ~ChunkedArray.null_count ~ChunkedArray.num_chunks + ~ChunkedArray.type \ No newline at end of file http://git-wip-us.apache.org/repos/asf/arrow-site/blob/4d4a3202/docs/python/_sources/generated/pyarrow.HadoopFileSystem.cat.rst.txt ---------------------------------------------------------------------- diff --git a/docs/python/_sources/generated/pyarrow.HadoopFileSystem.cat.rst.txt b/docs/python/_sources/generated/pyarrow.HadoopFileSystem.cat.rst.txt new file mode 100644 index 0000000..4223449 --- /dev/null +++ b/docs/python/_sources/generated/pyarrow.HadoopFileSystem.cat.rst.txt @@ -0,0 +1,6 @@ +pyarrow\.HadoopFileSystem\.cat +============================== + +.. currentmodule:: pyarrow + +.. automethod:: HadoopFileSystem.cat \ No newline at end of file http://git-wip-us.apache.org/repos/asf/arrow-site/blob/4d4a3202/docs/python/_sources/generated/pyarrow.HadoopFileSystem.chmod.rst.txt ---------------------------------------------------------------------- diff --git a/docs/python/_sources/generated/pyarrow.HadoopFileSystem.chmod.rst.txt b/docs/python/_sources/generated/pyarrow.HadoopFileSystem.chmod.rst.txt new file mode 100644 index 0000000..a0f0254 --- /dev/null +++ b/docs/python/_sources/generated/pyarrow.HadoopFileSystem.chmod.rst.txt @@ -0,0 +1,6 @@ +pyarrow\.HadoopFileSystem\.chmod +================================ + +.. currentmodule:: pyarrow + +.. automethod:: HadoopFileSystem.chmod \ No newline at end of file http://git-wip-us.apache.org/repos/asf/arrow-site/blob/4d4a3202/docs/python/_sources/generated/pyarrow.HadoopFileSystem.chown.rst.txt ---------------------------------------------------------------------- diff --git a/docs/python/_sources/generated/pyarrow.HadoopFileSystem.chown.rst.txt b/docs/python/_sources/generated/pyarrow.HadoopFileSystem.chown.rst.txt new file mode 100644 index 0000000..06b28f1 --- /dev/null +++ b/docs/python/_sources/generated/pyarrow.HadoopFileSystem.chown.rst.txt @@ -0,0 +1,6 @@ +pyarrow\.HadoopFileSystem\.chown +================================ + +.. currentmodule:: pyarrow + +.. automethod:: HadoopFileSystem.chown \ No newline at end of file http://git-wip-us.apache.org/repos/asf/arrow-site/blob/4d4a3202/docs/python/_sources/generated/pyarrow.HadoopFileSystem.delete.rst.txt ---------------------------------------------------------------------- diff --git a/docs/python/_sources/generated/pyarrow.HadoopFileSystem.delete.rst.txt b/docs/python/_sources/generated/pyarrow.HadoopFileSystem.delete.rst.txt new file mode 100644 index 0000000..ed1c0bc --- /dev/null +++ b/docs/python/_sources/generated/pyarrow.HadoopFileSystem.delete.rst.txt @@ -0,0 +1,6 @@ +pyarrow\.HadoopFileSystem\.delete +================================= + +.. currentmodule:: pyarrow + +.. automethod:: HadoopFileSystem.delete \ No newline at end of file http://git-wip-us.apache.org/repos/asf/arrow-site/blob/4d4a3202/docs/python/_sources/generated/pyarrow.HadoopFileSystem.df.rst.txt ---------------------------------------------------------------------- diff --git a/docs/python/_sources/generated/pyarrow.HadoopFileSystem.df.rst.txt b/docs/python/_sources/generated/pyarrow.HadoopFileSystem.df.rst.txt new file mode 100644 index 0000000..790e164 --- /dev/null +++ b/docs/python/_sources/generated/pyarrow.HadoopFileSystem.df.rst.txt @@ -0,0 +1,6 @@ +pyarrow\.HadoopFileSystem\.df +============================= + +.. currentmodule:: pyarrow + +.. automethod:: HadoopFileSystem.df \ No newline at end of file http://git-wip-us.apache.org/repos/asf/arrow-site/blob/4d4a3202/docs/python/_sources/generated/pyarrow.HadoopFileSystem.disk_usage.rst.txt ---------------------------------------------------------------------- diff --git a/docs/python/_sources/generated/pyarrow.HadoopFileSystem.disk_usage.rst.txt b/docs/python/_sources/generated/pyarrow.HadoopFileSystem.disk_usage.rst.txt new file mode 100644 index 0000000..3d35f8b --- /dev/null +++ b/docs/python/_sources/generated/pyarrow.HadoopFileSystem.disk_usage.rst.txt @@ -0,0 +1,6 @@ +pyarrow\.HadoopFileSystem\.disk\_usage +====================================== + +.. currentmodule:: pyarrow + +.. automethod:: HadoopFileSystem.disk_usage \ No newline at end of file http://git-wip-us.apache.org/repos/asf/arrow-site/blob/4d4a3202/docs/python/_sources/generated/pyarrow.HadoopFileSystem.download.rst.txt ---------------------------------------------------------------------- diff --git a/docs/python/_sources/generated/pyarrow.HadoopFileSystem.download.rst.txt b/docs/python/_sources/generated/pyarrow.HadoopFileSystem.download.rst.txt new file mode 100644 index 0000000..66c782c --- /dev/null +++ b/docs/python/_sources/generated/pyarrow.HadoopFileSystem.download.rst.txt @@ -0,0 +1,6 @@ +pyarrow\.HadoopFileSystem\.download +=================================== + +.. currentmodule:: pyarrow + +.. automethod:: HadoopFileSystem.download \ No newline at end of file http://git-wip-us.apache.org/repos/asf/arrow-site/blob/4d4a3202/docs/python/_sources/generated/pyarrow.HadoopFileSystem.exists.rst.txt ---------------------------------------------------------------------- diff --git a/docs/python/_sources/generated/pyarrow.HadoopFileSystem.exists.rst.txt b/docs/python/_sources/generated/pyarrow.HadoopFileSystem.exists.rst.txt new file mode 100644 index 0000000..201b491 --- /dev/null +++ b/docs/python/_sources/generated/pyarrow.HadoopFileSystem.exists.rst.txt @@ -0,0 +1,6 @@ +pyarrow\.HadoopFileSystem\.exists +================================= + +.. currentmodule:: pyarrow + +.. automethod:: HadoopFileSystem.exists \ No newline at end of file http://git-wip-us.apache.org/repos/asf/arrow-site/blob/4d4a3202/docs/python/_sources/generated/pyarrow.HadoopFileSystem.get_capacity.rst.txt ---------------------------------------------------------------------- diff --git a/docs/python/_sources/generated/pyarrow.HadoopFileSystem.get_capacity.rst.txt b/docs/python/_sources/generated/pyarrow.HadoopFileSystem.get_capacity.rst.txt new file mode 100644 index 0000000..e61af29 --- /dev/null +++ b/docs/python/_sources/generated/pyarrow.HadoopFileSystem.get_capacity.rst.txt @@ -0,0 +1,6 @@ +pyarrow\.HadoopFileSystem\.get\_capacity +======================================== + +.. currentmodule:: pyarrow + +.. automethod:: HadoopFileSystem.get_capacity \ No newline at end of file http://git-wip-us.apache.org/repos/asf/arrow-site/blob/4d4a3202/docs/python/_sources/generated/pyarrow.HadoopFileSystem.get_space_used.rst.txt ---------------------------------------------------------------------- diff --git a/docs/python/_sources/generated/pyarrow.HadoopFileSystem.get_space_used.rst.txt b/docs/python/_sources/generated/pyarrow.HadoopFileSystem.get_space_used.rst.txt new file mode 100644 index 0000000..35965c3 --- /dev/null +++ b/docs/python/_sources/generated/pyarrow.HadoopFileSystem.get_space_used.rst.txt @@ -0,0 +1,6 @@ +pyarrow\.HadoopFileSystem\.get\_space\_used +=========================================== + +.. currentmodule:: pyarrow + +.. automethod:: HadoopFileSystem.get_space_used \ No newline at end of file http://git-wip-us.apache.org/repos/asf/arrow-site/blob/4d4a3202/docs/python/_sources/generated/pyarrow.HadoopFileSystem.info.rst.txt ---------------------------------------------------------------------- diff --git a/docs/python/_sources/generated/pyarrow.HadoopFileSystem.info.rst.txt b/docs/python/_sources/generated/pyarrow.HadoopFileSystem.info.rst.txt new file mode 100644 index 0000000..02231ff --- /dev/null +++ b/docs/python/_sources/generated/pyarrow.HadoopFileSystem.info.rst.txt @@ -0,0 +1,6 @@ +pyarrow\.HadoopFileSystem\.info +=============================== + +.. currentmodule:: pyarrow + +.. automethod:: HadoopFileSystem.info \ No newline at end of file http://git-wip-us.apache.org/repos/asf/arrow-site/blob/4d4a3202/docs/python/_sources/generated/pyarrow.HadoopFileSystem.ls.rst.txt ---------------------------------------------------------------------- diff --git a/docs/python/_sources/generated/pyarrow.HadoopFileSystem.ls.rst.txt b/docs/python/_sources/generated/pyarrow.HadoopFileSystem.ls.rst.txt new file mode 100644 index 0000000..ff5b592 --- /dev/null +++ b/docs/python/_sources/generated/pyarrow.HadoopFileSystem.ls.rst.txt @@ -0,0 +1,6 @@ +pyarrow\.HadoopFileSystem\.ls +============================= + +.. currentmodule:: pyarrow + +.. automethod:: HadoopFileSystem.ls \ No newline at end of file http://git-wip-us.apache.org/repos/asf/arrow-site/blob/4d4a3202/docs/python/_sources/generated/pyarrow.HadoopFileSystem.mkdir.rst.txt ---------------------------------------------------------------------- diff --git a/docs/python/_sources/generated/pyarrow.HadoopFileSystem.mkdir.rst.txt b/docs/python/_sources/generated/pyarrow.HadoopFileSystem.mkdir.rst.txt new file mode 100644 index 0000000..a2b4476 --- /dev/null +++ b/docs/python/_sources/generated/pyarrow.HadoopFileSystem.mkdir.rst.txt @@ -0,0 +1,6 @@ +pyarrow\.HadoopFileSystem\.mkdir +================================ + +.. currentmodule:: pyarrow + +.. automethod:: HadoopFileSystem.mkdir \ No newline at end of file http://git-wip-us.apache.org/repos/asf/arrow-site/blob/4d4a3202/docs/python/_sources/generated/pyarrow.HadoopFileSystem.open.rst.txt ---------------------------------------------------------------------- diff --git a/docs/python/_sources/generated/pyarrow.HadoopFileSystem.open.rst.txt b/docs/python/_sources/generated/pyarrow.HadoopFileSystem.open.rst.txt new file mode 100644 index 0000000..8b65982 --- /dev/null +++ b/docs/python/_sources/generated/pyarrow.HadoopFileSystem.open.rst.txt @@ -0,0 +1,6 @@ +pyarrow\.HadoopFileSystem\.open +=============================== + +.. currentmodule:: pyarrow + +.. automethod:: HadoopFileSystem.open \ No newline at end of file http://git-wip-us.apache.org/repos/asf/arrow-site/blob/4d4a3202/docs/python/_sources/generated/pyarrow.HadoopFileSystem.rename.rst.txt ---------------------------------------------------------------------- diff --git a/docs/python/_sources/generated/pyarrow.HadoopFileSystem.rename.rst.txt b/docs/python/_sources/generated/pyarrow.HadoopFileSystem.rename.rst.txt new file mode 100644 index 0000000..bc04211 --- /dev/null +++ b/docs/python/_sources/generated/pyarrow.HadoopFileSystem.rename.rst.txt @@ -0,0 +1,6 @@ +pyarrow\.HadoopFileSystem\.rename +================================= + +.. currentmodule:: pyarrow + +.. automethod:: HadoopFileSystem.rename \ No newline at end of file http://git-wip-us.apache.org/repos/asf/arrow-site/blob/4d4a3202/docs/python/_sources/generated/pyarrow.HadoopFileSystem.rm.rst.txt ---------------------------------------------------------------------- diff --git a/docs/python/_sources/generated/pyarrow.HadoopFileSystem.rm.rst.txt b/docs/python/_sources/generated/pyarrow.HadoopFileSystem.rm.rst.txt new file mode 100644 index 0000000..f339863 --- /dev/null +++ b/docs/python/_sources/generated/pyarrow.HadoopFileSystem.rm.rst.txt @@ -0,0 +1,6 @@ +pyarrow\.HadoopFileSystem\.rm +============================= + +.. currentmodule:: pyarrow + +.. automethod:: HadoopFileSystem.rm \ No newline at end of file http://git-wip-us.apache.org/repos/asf/arrow-site/blob/4d4a3202/docs/python/_sources/generated/pyarrow.HadoopFileSystem.upload.rst.txt ---------------------------------------------------------------------- diff --git a/docs/python/_sources/generated/pyarrow.HadoopFileSystem.upload.rst.txt b/docs/python/_sources/generated/pyarrow.HadoopFileSystem.upload.rst.txt new file mode 100644 index 0000000..30d1c27 --- /dev/null +++ b/docs/python/_sources/generated/pyarrow.HadoopFileSystem.upload.rst.txt @@ -0,0 +1,6 @@ +pyarrow\.HadoopFileSystem\.upload +================================= + +.. currentmodule:: pyarrow + +.. automethod:: HadoopFileSystem.upload \ No newline at end of file http://git-wip-us.apache.org/repos/asf/arrow-site/blob/4d4a3202/docs/python/_sources/generated/pyarrow.HdfsFile.rst.txt ---------------------------------------------------------------------- diff --git a/docs/python/_sources/generated/pyarrow.HdfsFile.rst.txt b/docs/python/_sources/generated/pyarrow.HdfsFile.rst.txt new file mode 100644 index 0000000..9facf14 --- /dev/null +++ b/docs/python/_sources/generated/pyarrow.HdfsFile.rst.txt @@ -0,0 +1,38 @@ +pyarrow\.HdfsFile +================= + +.. currentmodule:: pyarrow + +.. autoclass:: HdfsFile + + + .. automethod:: __init__ + + + .. rubric:: Methods + + .. autosummary:: + + ~HdfsFile.close + ~HdfsFile.download + ~HdfsFile.read + ~HdfsFile.read_buffer + ~HdfsFile.seek + ~HdfsFile.size + ~HdfsFile.tell + ~HdfsFile.upload + ~HdfsFile.write + + + + + + .. rubric:: Attributes + + .. autosummary:: + + ~HdfsFile.buffer_size + ~HdfsFile.mode + ~HdfsFile.parent + + \ No newline at end of file http://git-wip-us.apache.org/repos/asf/arrow-site/blob/4d4a3202/docs/python/_sources/generated/pyarrow.LocalFileSystem.rst.txt ---------------------------------------------------------------------- diff --git a/docs/python/_sources/generated/pyarrow.LocalFileSystem.rst.txt b/docs/python/_sources/generated/pyarrow.LocalFileSystem.rst.txt new file mode 100644 index 0000000..70b2ea7 --- /dev/null +++ b/docs/python/_sources/generated/pyarrow.LocalFileSystem.rst.txt @@ -0,0 +1,43 @@ +pyarrow\.LocalFileSystem +======================== + +.. currentmodule:: pyarrow + +.. autoclass:: LocalFileSystem + + + .. automethod:: __init__ + + + .. rubric:: Methods + + .. autosummary:: + + ~LocalFileSystem.cat + ~LocalFileSystem.delete + ~LocalFileSystem.disk_usage + ~LocalFileSystem.exists + ~LocalFileSystem.get_instance + ~LocalFileSystem.isdir + ~LocalFileSystem.isfile + ~LocalFileSystem.ls + ~LocalFileSystem.mkdir + ~LocalFileSystem.mv + ~LocalFileSystem.open + ~LocalFileSystem.read_parquet + ~LocalFileSystem.rename + ~LocalFileSystem.rm + ~LocalFileSystem.stat + ~LocalFileSystem.walk + + + + + + .. rubric:: Attributes + + .. autosummary:: + + ~LocalFileSystem.pathsep + + \ No newline at end of file http://git-wip-us.apache.org/repos/asf/arrow-site/blob/4d4a3202/docs/python/_sources/generated/pyarrow.RecordBatch.rst.txt ---------------------------------------------------------------------- diff --git a/docs/python/_sources/generated/pyarrow.RecordBatch.rst.txt b/docs/python/_sources/generated/pyarrow.RecordBatch.rst.txt index ff32084..16d5334 100644 --- a/docs/python/_sources/generated/pyarrow.RecordBatch.rst.txt +++ b/docs/python/_sources/generated/pyarrow.RecordBatch.rst.txt @@ -17,6 +17,7 @@ pyarrow\.RecordBatch ~RecordBatch.equals ~RecordBatch.from_arrays ~RecordBatch.from_pandas + ~RecordBatch.replace_schema_metadata ~RecordBatch.slice ~RecordBatch.to_pandas ~RecordBatch.to_pydict http://git-wip-us.apache.org/repos/asf/arrow-site/blob/4d4a3202/docs/python/_sources/generated/pyarrow.Table.rst.txt ---------------------------------------------------------------------- diff --git a/docs/python/_sources/generated/pyarrow.Table.rst.txt b/docs/python/_sources/generated/pyarrow.Table.rst.txt index 996d5d8..6818041 100644 --- a/docs/python/_sources/generated/pyarrow.Table.rst.txt +++ b/docs/python/_sources/generated/pyarrow.Table.rst.txt @@ -22,6 +22,7 @@ pyarrow\.Table ~Table.from_pandas ~Table.itercolumns ~Table.remove_column + ~Table.replace_schema_metadata ~Table.to_pandas ~Table.to_pydict http://git-wip-us.apache.org/repos/asf/arrow-site/blob/4d4a3202/docs/python/_sources/generated/pyarrow.hdfs.connect.rst.txt ---------------------------------------------------------------------- diff --git a/docs/python/_sources/generated/pyarrow.hdfs.connect.rst.txt b/docs/python/_sources/generated/pyarrow.hdfs.connect.rst.txt new file mode 100644 index 0000000..a188990 --- /dev/null +++ b/docs/python/_sources/generated/pyarrow.hdfs.connect.rst.txt @@ -0,0 +1,6 @@ +pyarrow\.hdfs\.connect +====================== + +.. currentmodule:: pyarrow.hdfs + +.. autofunction:: connect \ No newline at end of file http://git-wip-us.apache.org/repos/asf/arrow-site/blob/4d4a3202/docs/python/_sources/generated/pyarrow.lib.Array.rst.txt ---------------------------------------------------------------------- diff --git a/docs/python/_sources/generated/pyarrow.lib.Array.rst.txt b/docs/python/_sources/generated/pyarrow.lib.Array.rst.txt new file mode 100644 index 0000000..35445b5 --- /dev/null +++ b/docs/python/_sources/generated/pyarrow.lib.Array.rst.txt @@ -0,0 +1,35 @@ +pyarrow\.lib\.Array +=================== + +.. currentmodule:: pyarrow.lib + +.. autoclass:: Array + + + .. automethod:: __init__ + + + .. rubric:: Methods + + .. autosummary:: + + ~Array.equals + ~Array.from_pandas + ~Array.isnull + ~Array.slice + ~Array.to_pandas + ~Array.to_pylist + ~Array.validate + + + + + + .. rubric:: Attributes + + .. autosummary:: + + ~Array.null_count + ~Array.type + + \ No newline at end of file http://git-wip-us.apache.org/repos/asf/arrow-site/blob/4d4a3202/docs/python/_sources/generated/pyarrow.lib.BinaryArray.rst.txt ---------------------------------------------------------------------- diff --git a/docs/python/_sources/generated/pyarrow.lib.BinaryArray.rst.txt b/docs/python/_sources/generated/pyarrow.lib.BinaryArray.rst.txt new file mode 100644 index 0000000..4d7f7e2 --- /dev/null +++ b/docs/python/_sources/generated/pyarrow.lib.BinaryArray.rst.txt @@ -0,0 +1,35 @@ +pyarrow\.lib\.BinaryArray +========================= + +.. currentmodule:: pyarrow.lib + +.. autoclass:: BinaryArray + + + .. automethod:: __init__ + + + .. rubric:: Methods + + .. autosummary:: + + ~BinaryArray.equals + ~BinaryArray.from_pandas + ~BinaryArray.isnull + ~BinaryArray.slice + ~BinaryArray.to_pandas + ~BinaryArray.to_pylist + ~BinaryArray.validate + + + + + + .. rubric:: Attributes + + .. autosummary:: + + ~BinaryArray.null_count + ~BinaryArray.type + + \ No newline at end of file http://git-wip-us.apache.org/repos/asf/arrow-site/blob/4d4a3202/docs/python/_sources/generated/pyarrow.lib.BooleanArray.rst.txt ---------------------------------------------------------------------- diff --git a/docs/python/_sources/generated/pyarrow.lib.BooleanArray.rst.txt b/docs/python/_sources/generated/pyarrow.lib.BooleanArray.rst.txt new file mode 100644 index 0000000..51e9b41 --- /dev/null +++ b/docs/python/_sources/generated/pyarrow.lib.BooleanArray.rst.txt @@ -0,0 +1,35 @@ +pyarrow\.lib\.BooleanArray +========================== + +.. currentmodule:: pyarrow.lib + +.. autoclass:: BooleanArray + + + .. automethod:: __init__ + + + .. rubric:: Methods + + .. autosummary:: + + ~BooleanArray.equals + ~BooleanArray.from_pandas + ~BooleanArray.isnull + ~BooleanArray.slice + ~BooleanArray.to_pandas + ~BooleanArray.to_pylist + ~BooleanArray.validate + + + + + + .. rubric:: Attributes + + .. autosummary:: + + ~BooleanArray.null_count + ~BooleanArray.type + + \ No newline at end of file http://git-wip-us.apache.org/repos/asf/arrow-site/blob/4d4a3202/docs/python/_sources/generated/pyarrow.lib.DataType.rst.txt ---------------------------------------------------------------------- diff --git a/docs/python/_sources/generated/pyarrow.lib.DataType.rst.txt b/docs/python/_sources/generated/pyarrow.lib.DataType.rst.txt new file mode 100644 index 0000000..a678796 --- /dev/null +++ b/docs/python/_sources/generated/pyarrow.lib.DataType.rst.txt @@ -0,0 +1,28 @@ +pyarrow\.lib\.DataType +====================== + +.. currentmodule:: pyarrow.lib + +.. autoclass:: DataType + + + .. automethod:: __init__ + + + .. rubric:: Methods + + .. autosummary:: + + ~DataType.to_pandas_dtype + + + + + + .. rubric:: Attributes + + .. autosummary:: + + ~DataType.id + + \ No newline at end of file http://git-wip-us.apache.org/repos/asf/arrow-site/blob/4d4a3202/docs/python/_sources/generated/pyarrow.lib.Date32Array.rst.txt ---------------------------------------------------------------------- diff --git a/docs/python/_sources/generated/pyarrow.lib.Date32Array.rst.txt b/docs/python/_sources/generated/pyarrow.lib.Date32Array.rst.txt new file mode 100644 index 0000000..2e76599 --- /dev/null +++ b/docs/python/_sources/generated/pyarrow.lib.Date32Array.rst.txt @@ -0,0 +1,35 @@ +pyarrow\.lib\.Date32Array +========================= + +.. currentmodule:: pyarrow.lib + +.. autoclass:: Date32Array + + + .. automethod:: __init__ + + + .. rubric:: Methods + + .. autosummary:: + + ~Date32Array.equals + ~Date32Array.from_pandas + ~Date32Array.isnull + ~Date32Array.slice + ~Date32Array.to_pandas + ~Date32Array.to_pylist + ~Date32Array.validate + + + + + + .. rubric:: Attributes + + .. autosummary:: + + ~Date32Array.null_count + ~Date32Array.type + + \ No newline at end of file http://git-wip-us.apache.org/repos/asf/arrow-site/blob/4d4a3202/docs/python/_sources/generated/pyarrow.lib.Date64Array.rst.txt ---------------------------------------------------------------------- diff --git a/docs/python/_sources/generated/pyarrow.lib.Date64Array.rst.txt b/docs/python/_sources/generated/pyarrow.lib.Date64Array.rst.txt new file mode 100644 index 0000000..69ecfac --- /dev/null +++ b/docs/python/_sources/generated/pyarrow.lib.Date64Array.rst.txt @@ -0,0 +1,35 @@ +pyarrow\.lib\.Date64Array +========================= + +.. currentmodule:: pyarrow.lib + +.. autoclass:: Date64Array + + + .. automethod:: __init__ + + + .. rubric:: Methods + + .. autosummary:: + + ~Date64Array.equals + ~Date64Array.from_pandas + ~Date64Array.isnull + ~Date64Array.slice + ~Date64Array.to_pandas + ~Date64Array.to_pylist + ~Date64Array.validate + + + + + + .. rubric:: Attributes + + .. autosummary:: + + ~Date64Array.null_count + ~Date64Array.type + + \ No newline at end of file http://git-wip-us.apache.org/repos/asf/arrow-site/blob/4d4a3202/docs/python/_sources/generated/pyarrow.lib.DecimalArray.rst.txt ---------------------------------------------------------------------- diff --git a/docs/python/_sources/generated/pyarrow.lib.DecimalArray.rst.txt b/docs/python/_sources/generated/pyarrow.lib.DecimalArray.rst.txt new file mode 100644 index 0000000..57aa92d --- /dev/null +++ b/docs/python/_sources/generated/pyarrow.lib.DecimalArray.rst.txt @@ -0,0 +1,35 @@ +pyarrow\.lib\.DecimalArray +========================== + +.. currentmodule:: pyarrow.lib + +.. autoclass:: DecimalArray + + + .. automethod:: __init__ + + + .. rubric:: Methods + + .. autosummary:: + + ~DecimalArray.equals + ~DecimalArray.from_pandas + ~DecimalArray.isnull + ~DecimalArray.slice + ~DecimalArray.to_pandas + ~DecimalArray.to_pylist + ~DecimalArray.validate + + + + + + .. rubric:: Attributes + + .. autosummary:: + + ~DecimalArray.null_count + ~DecimalArray.type + + \ No newline at end of file http://git-wip-us.apache.org/repos/asf/arrow-site/blob/4d4a3202/docs/python/_sources/generated/pyarrow.lib.DictionaryArray.rst.txt ---------------------------------------------------------------------- diff --git a/docs/python/_sources/generated/pyarrow.lib.DictionaryArray.rst.txt b/docs/python/_sources/generated/pyarrow.lib.DictionaryArray.rst.txt new file mode 100644 index 0000000..b5a724c --- /dev/null +++ b/docs/python/_sources/generated/pyarrow.lib.DictionaryArray.rst.txt @@ -0,0 +1,38 @@ +pyarrow\.lib\.DictionaryArray +============================= + +.. currentmodule:: pyarrow.lib + +.. autoclass:: DictionaryArray + + + .. automethod:: __init__ + + + .. rubric:: Methods + + .. autosummary:: + + ~DictionaryArray.equals + ~DictionaryArray.from_arrays + ~DictionaryArray.from_pandas + ~DictionaryArray.isnull + ~DictionaryArray.slice + ~DictionaryArray.to_pandas + ~DictionaryArray.to_pylist + ~DictionaryArray.validate + + + + + + .. rubric:: Attributes + + .. autosummary:: + + ~DictionaryArray.dictionary + ~DictionaryArray.indices + ~DictionaryArray.null_count + ~DictionaryArray.type + + \ No newline at end of file http://git-wip-us.apache.org/repos/asf/arrow-site/blob/4d4a3202/docs/python/_sources/generated/pyarrow.lib.Field.rst.txt ---------------------------------------------------------------------- diff --git a/docs/python/_sources/generated/pyarrow.lib.Field.rst.txt b/docs/python/_sources/generated/pyarrow.lib.Field.rst.txt new file mode 100644 index 0000000..930e4f6 --- /dev/null +++ b/docs/python/_sources/generated/pyarrow.lib.Field.rst.txt @@ -0,0 +1,33 @@ +pyarrow\.lib\.Field +=================== + +.. currentmodule:: pyarrow.lib + +.. autoclass:: Field + + + .. automethod:: __init__ + + + .. rubric:: Methods + + .. autosummary:: + + ~Field.add_metadata + ~Field.equals + ~Field.remove_metadata + + + + + + .. rubric:: Attributes + + .. autosummary:: + + ~Field.metadata + ~Field.name + ~Field.nullable + ~Field.type + + \ No newline at end of file http://git-wip-us.apache.org/repos/asf/arrow-site/blob/4d4a3202/docs/python/_sources/generated/pyarrow.lib.FixedSizeBinaryArray.rst.txt ---------------------------------------------------------------------- diff --git a/docs/python/_sources/generated/pyarrow.lib.FixedSizeBinaryArray.rst.txt b/docs/python/_sources/generated/pyarrow.lib.FixedSizeBinaryArray.rst.txt new file mode 100644 index 0000000..5617aac --- /dev/null +++ b/docs/python/_sources/generated/pyarrow.lib.FixedSizeBinaryArray.rst.txt @@ -0,0 +1,35 @@ +pyarrow\.lib\.FixedSizeBinaryArray +================================== + +.. currentmodule:: pyarrow.lib + +.. autoclass:: FixedSizeBinaryArray + + + .. automethod:: __init__ + + + .. rubric:: Methods + + .. autosummary:: + + ~FixedSizeBinaryArray.equals + ~FixedSizeBinaryArray.from_pandas + ~FixedSizeBinaryArray.isnull + ~FixedSizeBinaryArray.slice + ~FixedSizeBinaryArray.to_pandas + ~FixedSizeBinaryArray.to_pylist + ~FixedSizeBinaryArray.validate + + + + + + .. rubric:: Attributes + + .. autosummary:: + + ~FixedSizeBinaryArray.null_count + ~FixedSizeBinaryArray.type + + \ No newline at end of file http://git-wip-us.apache.org/repos/asf/arrow-site/blob/4d4a3202/docs/python/_sources/generated/pyarrow.lib.FloatingPointArray.rst.txt ---------------------------------------------------------------------- diff --git a/docs/python/_sources/generated/pyarrow.lib.FloatingPointArray.rst.txt b/docs/python/_sources/generated/pyarrow.lib.FloatingPointArray.rst.txt new file mode 100644 index 0000000..be3ed52 --- /dev/null +++ b/docs/python/_sources/generated/pyarrow.lib.FloatingPointArray.rst.txt @@ -0,0 +1,35 @@ +pyarrow\.lib\.FloatingPointArray +================================ + +.. currentmodule:: pyarrow.lib + +.. autoclass:: FloatingPointArray + + + .. automethod:: __init__ + + + .. rubric:: Methods + + .. autosummary:: + + ~FloatingPointArray.equals + ~FloatingPointArray.from_pandas + ~FloatingPointArray.isnull + ~FloatingPointArray.slice + ~FloatingPointArray.to_pandas + ~FloatingPointArray.to_pylist + ~FloatingPointArray.validate + + + + + + .. rubric:: Attributes + + .. autosummary:: + + ~FloatingPointArray.null_count + ~FloatingPointArray.type + + \ No newline at end of file http://git-wip-us.apache.org/repos/asf/arrow-site/blob/4d4a3202/docs/python/_sources/generated/pyarrow.lib.Int16Array.rst.txt ---------------------------------------------------------------------- diff --git a/docs/python/_sources/generated/pyarrow.lib.Int16Array.rst.txt b/docs/python/_sources/generated/pyarrow.lib.Int16Array.rst.txt new file mode 100644 index 0000000..5c57c6a --- /dev/null +++ b/docs/python/_sources/generated/pyarrow.lib.Int16Array.rst.txt @@ -0,0 +1,35 @@ +pyarrow\.lib\.Int16Array +======================== + +.. currentmodule:: pyarrow.lib + +.. autoclass:: Int16Array + + + .. automethod:: __init__ + + + .. rubric:: Methods + + .. autosummary:: + + ~Int16Array.equals + ~Int16Array.from_pandas + ~Int16Array.isnull + ~Int16Array.slice + ~Int16Array.to_pandas + ~Int16Array.to_pylist + ~Int16Array.validate + + + + + + .. rubric:: Attributes + + .. autosummary:: + + ~Int16Array.null_count + ~Int16Array.type + + \ No newline at end of file http://git-wip-us.apache.org/repos/asf/arrow-site/blob/4d4a3202/docs/python/_sources/generated/pyarrow.lib.Int32Array.rst.txt ---------------------------------------------------------------------- diff --git a/docs/python/_sources/generated/pyarrow.lib.Int32Array.rst.txt b/docs/python/_sources/generated/pyarrow.lib.Int32Array.rst.txt new file mode 100644 index 0000000..b63d745 --- /dev/null +++ b/docs/python/_sources/generated/pyarrow.lib.Int32Array.rst.txt @@ -0,0 +1,35 @@ +pyarrow\.lib\.Int32Array +======================== + +.. currentmodule:: pyarrow.lib + +.. autoclass:: Int32Array + + + .. automethod:: __init__ + + + .. rubric:: Methods + + .. autosummary:: + + ~Int32Array.equals + ~Int32Array.from_pandas + ~Int32Array.isnull + ~Int32Array.slice + ~Int32Array.to_pandas + ~Int32Array.to_pylist + ~Int32Array.validate + + + + + + .. rubric:: Attributes + + .. autosummary:: + + ~Int32Array.null_count + ~Int32Array.type + + \ No newline at end of file http://git-wip-us.apache.org/repos/asf/arrow-site/blob/4d4a3202/docs/python/_sources/generated/pyarrow.lib.Int64Array.rst.txt ---------------------------------------------------------------------- diff --git a/docs/python/_sources/generated/pyarrow.lib.Int64Array.rst.txt b/docs/python/_sources/generated/pyarrow.lib.Int64Array.rst.txt new file mode 100644 index 0000000..d67cd4a --- /dev/null +++ b/docs/python/_sources/generated/pyarrow.lib.Int64Array.rst.txt @@ -0,0 +1,35 @@ +pyarrow\.lib\.Int64Array +======================== + +.. currentmodule:: pyarrow.lib + +.. autoclass:: Int64Array + + + .. automethod:: __init__ + + + .. rubric:: Methods + + .. autosummary:: + + ~Int64Array.equals + ~Int64Array.from_pandas + ~Int64Array.isnull + ~Int64Array.slice + ~Int64Array.to_pandas + ~Int64Array.to_pylist + ~Int64Array.validate + + + + + + .. rubric:: Attributes + + .. autosummary:: + + ~Int64Array.null_count + ~Int64Array.type + + \ No newline at end of file http://git-wip-us.apache.org/repos/asf/arrow-site/blob/4d4a3202/docs/python/_sources/generated/pyarrow.lib.Int8Array.rst.txt ---------------------------------------------------------------------- diff --git a/docs/python/_sources/generated/pyarrow.lib.Int8Array.rst.txt b/docs/python/_sources/generated/pyarrow.lib.Int8Array.rst.txt new file mode 100644 index 0000000..277eea7 --- /dev/null +++ b/docs/python/_sources/generated/pyarrow.lib.Int8Array.rst.txt @@ -0,0 +1,35 @@ +pyarrow\.lib\.Int8Array +======================= + +.. currentmodule:: pyarrow.lib + +.. autoclass:: Int8Array + + + .. automethod:: __init__ + + + .. rubric:: Methods + + .. autosummary:: + + ~Int8Array.equals + ~Int8Array.from_pandas + ~Int8Array.isnull + ~Int8Array.slice + ~Int8Array.to_pandas + ~Int8Array.to_pylist + ~Int8Array.validate + + + + + + .. rubric:: Attributes + + .. autosummary:: + + ~Int8Array.null_count + ~Int8Array.type + + \ No newline at end of file http://git-wip-us.apache.org/repos/asf/arrow-site/blob/4d4a3202/docs/python/_sources/generated/pyarrow.lib.IntegerArray.rst.txt ---------------------------------------------------------------------- diff --git a/docs/python/_sources/generated/pyarrow.lib.IntegerArray.rst.txt b/docs/python/_sources/generated/pyarrow.lib.IntegerArray.rst.txt new file mode 100644 index 0000000..6b38fc3 --- /dev/null +++ b/docs/python/_sources/generated/pyarrow.lib.IntegerArray.rst.txt @@ -0,0 +1,35 @@ +pyarrow\.lib\.IntegerArray +========================== + +.. currentmodule:: pyarrow.lib + +.. autoclass:: IntegerArray + + + .. automethod:: __init__ + + + .. rubric:: Methods + + .. autosummary:: + + ~IntegerArray.equals + ~IntegerArray.from_pandas + ~IntegerArray.isnull + ~IntegerArray.slice + ~IntegerArray.to_pandas + ~IntegerArray.to_pylist + ~IntegerArray.validate + + + + + + .. rubric:: Attributes + + .. autosummary:: + + ~IntegerArray.null_count + ~IntegerArray.type + + \ No newline at end of file http://git-wip-us.apache.org/repos/asf/arrow-site/blob/4d4a3202/docs/python/_sources/generated/pyarrow.lib.ListArray.rst.txt ---------------------------------------------------------------------- diff --git a/docs/python/_sources/generated/pyarrow.lib.ListArray.rst.txt b/docs/python/_sources/generated/pyarrow.lib.ListArray.rst.txt new file mode 100644 index 0000000..1ca5075 --- /dev/null +++ b/docs/python/_sources/generated/pyarrow.lib.ListArray.rst.txt @@ -0,0 +1,36 @@ +pyarrow\.lib\.ListArray +======================= + +.. currentmodule:: pyarrow.lib + +.. autoclass:: ListArray + + + .. automethod:: __init__ + + + .. rubric:: Methods + + .. autosummary:: + + ~ListArray.equals + ~ListArray.from_arrays + ~ListArray.from_pandas + ~ListArray.isnull + ~ListArray.slice + ~ListArray.to_pandas + ~ListArray.to_pylist + ~ListArray.validate + + + + + + .. rubric:: Attributes + + .. autosummary:: + + ~ListArray.null_count + ~ListArray.type + + \ No newline at end of file http://git-wip-us.apache.org/repos/asf/arrow-site/blob/4d4a3202/docs/python/_sources/generated/pyarrow.lib.NullArray.rst.txt ---------------------------------------------------------------------- diff --git a/docs/python/_sources/generated/pyarrow.lib.NullArray.rst.txt b/docs/python/_sources/generated/pyarrow.lib.NullArray.rst.txt new file mode 100644 index 0000000..da87211 --- /dev/null +++ b/docs/python/_sources/generated/pyarrow.lib.NullArray.rst.txt @@ -0,0 +1,35 @@ +pyarrow\.lib\.NullArray +======================= + +.. currentmodule:: pyarrow.lib + +.. autoclass:: NullArray + + + .. automethod:: __init__ + + + .. rubric:: Methods + + .. autosummary:: + + ~NullArray.equals + ~NullArray.from_pandas + ~NullArray.isnull + ~NullArray.slice + ~NullArray.to_pandas + ~NullArray.to_pylist + ~NullArray.validate + + + + + + .. rubric:: Attributes + + .. autosummary:: + + ~NullArray.null_count + ~NullArray.type + + \ No newline at end of file http://git-wip-us.apache.org/repos/asf/arrow-site/blob/4d4a3202/docs/python/_sources/generated/pyarrow.lib.NumericArray.rst.txt ---------------------------------------------------------------------- diff --git a/docs/python/_sources/generated/pyarrow.lib.NumericArray.rst.txt b/docs/python/_sources/generated/pyarrow.lib.NumericArray.rst.txt new file mode 100644 index 0000000..45ef33c --- /dev/null +++ b/docs/python/_sources/generated/pyarrow.lib.NumericArray.rst.txt @@ -0,0 +1,35 @@ +pyarrow\.lib\.NumericArray +========================== + +.. currentmodule:: pyarrow.lib + +.. autoclass:: NumericArray + + + .. automethod:: __init__ + + + .. rubric:: Methods + + .. autosummary:: + + ~NumericArray.equals + ~NumericArray.from_pandas + ~NumericArray.isnull + ~NumericArray.slice + ~NumericArray.to_pandas + ~NumericArray.to_pylist + ~NumericArray.validate + + + + + + .. rubric:: Attributes + + .. autosummary:: + + ~NumericArray.null_count + ~NumericArray.type + + \ No newline at end of file http://git-wip-us.apache.org/repos/asf/arrow-site/blob/4d4a3202/docs/python/_sources/generated/pyarrow.lib.Schema.rst.txt ---------------------------------------------------------------------- diff --git a/docs/python/_sources/generated/pyarrow.lib.Schema.rst.txt b/docs/python/_sources/generated/pyarrow.lib.Schema.rst.txt new file mode 100644 index 0000000..c8b2128 --- /dev/null +++ b/docs/python/_sources/generated/pyarrow.lib.Schema.rst.txt @@ -0,0 +1,33 @@ +pyarrow\.lib\.Schema +==================== + +.. currentmodule:: pyarrow.lib + +.. autoclass:: Schema + + + .. automethod:: __init__ + + + .. rubric:: Methods + + .. autosummary:: + + ~Schema.add_metadata + ~Schema.equals + ~Schema.field_by_name + ~Schema.get_field_index + ~Schema.remove_metadata + + + + + + .. rubric:: Attributes + + .. autosummary:: + + ~Schema.metadata + ~Schema.names + + \ No newline at end of file http://git-wip-us.apache.org/repos/asf/arrow-site/blob/4d4a3202/docs/python/_sources/generated/pyarrow.lib.StringArray.rst.txt ---------------------------------------------------------------------- diff --git a/docs/python/_sources/generated/pyarrow.lib.StringArray.rst.txt b/docs/python/_sources/generated/pyarrow.lib.StringArray.rst.txt new file mode 100644 index 0000000..046fd9f --- /dev/null +++ b/docs/python/_sources/generated/pyarrow.lib.StringArray.rst.txt @@ -0,0 +1,35 @@ +pyarrow\.lib\.StringArray +========================= + +.. currentmodule:: pyarrow.lib + +.. autoclass:: StringArray + + + .. automethod:: __init__ + + + .. rubric:: Methods + + .. autosummary:: + + ~StringArray.equals + ~StringArray.from_pandas + ~StringArray.isnull + ~StringArray.slice + ~StringArray.to_pandas + ~StringArray.to_pylist + ~StringArray.validate + + + + + + .. rubric:: Attributes + + .. autosummary:: + + ~StringArray.null_count + ~StringArray.type + + \ No newline at end of file http://git-wip-us.apache.org/repos/asf/arrow-site/blob/4d4a3202/docs/python/_sources/generated/pyarrow.lib.Time32Array.rst.txt ---------------------------------------------------------------------- diff --git a/docs/python/_sources/generated/pyarrow.lib.Time32Array.rst.txt b/docs/python/_sources/generated/pyarrow.lib.Time32Array.rst.txt new file mode 100644 index 0000000..0b22ec5 --- /dev/null +++ b/docs/python/_sources/generated/pyarrow.lib.Time32Array.rst.txt @@ -0,0 +1,35 @@ +pyarrow\.lib\.Time32Array +========================= + +.. currentmodule:: pyarrow.lib + +.. autoclass:: Time32Array + + + .. automethod:: __init__ + + + .. rubric:: Methods + + .. autosummary:: + + ~Time32Array.equals + ~Time32Array.from_pandas + ~Time32Array.isnull + ~Time32Array.slice + ~Time32Array.to_pandas + ~Time32Array.to_pylist + ~Time32Array.validate + + + + + + .. rubric:: Attributes + + .. autosummary:: + + ~Time32Array.null_count + ~Time32Array.type + + \ No newline at end of file http://git-wip-us.apache.org/repos/asf/arrow-site/blob/4d4a3202/docs/python/_sources/generated/pyarrow.lib.Time64Array.rst.txt ---------------------------------------------------------------------- diff --git a/docs/python/_sources/generated/pyarrow.lib.Time64Array.rst.txt b/docs/python/_sources/generated/pyarrow.lib.Time64Array.rst.txt new file mode 100644 index 0000000..d29ef46 --- /dev/null +++ b/docs/python/_sources/generated/pyarrow.lib.Time64Array.rst.txt @@ -0,0 +1,35 @@ +pyarrow\.lib\.Time64Array +========================= + +.. currentmodule:: pyarrow.lib + +.. autoclass:: Time64Array + + + .. automethod:: __init__ + + + .. rubric:: Methods + + .. autosummary:: + + ~Time64Array.equals + ~Time64Array.from_pandas + ~Time64Array.isnull + ~Time64Array.slice + ~Time64Array.to_pandas + ~Time64Array.to_pylist + ~Time64Array.validate + + + + + + .. rubric:: Attributes + + .. autosummary:: + + ~Time64Array.null_count + ~Time64Array.type + + \ No newline at end of file http://git-wip-us.apache.org/repos/asf/arrow-site/blob/4d4a3202/docs/python/_sources/generated/pyarrow.lib.TimestampArray.rst.txt ---------------------------------------------------------------------- diff --git a/docs/python/_sources/generated/pyarrow.lib.TimestampArray.rst.txt b/docs/python/_sources/generated/pyarrow.lib.TimestampArray.rst.txt new file mode 100644 index 0000000..a5eb4ff --- /dev/null +++ b/docs/python/_sources/generated/pyarrow.lib.TimestampArray.rst.txt @@ -0,0 +1,35 @@ +pyarrow\.lib\.TimestampArray +============================ + +.. currentmodule:: pyarrow.lib + +.. autoclass:: TimestampArray + + + .. automethod:: __init__ + + + .. rubric:: Methods + + .. autosummary:: + + ~TimestampArray.equals + ~TimestampArray.from_pandas + ~TimestampArray.isnull + ~TimestampArray.slice + ~TimestampArray.to_pandas + ~TimestampArray.to_pylist + ~TimestampArray.validate + + + + + + .. rubric:: Attributes + + .. autosummary:: + + ~TimestampArray.null_count + ~TimestampArray.type + + \ No newline at end of file http://git-wip-us.apache.org/repos/asf/arrow-site/blob/4d4a3202/docs/python/_sources/generated/pyarrow.lib.UInt16Array.rst.txt ---------------------------------------------------------------------- diff --git a/docs/python/_sources/generated/pyarrow.lib.UInt16Array.rst.txt b/docs/python/_sources/generated/pyarrow.lib.UInt16Array.rst.txt new file mode 100644 index 0000000..e7bd187 --- /dev/null +++ b/docs/python/_sources/generated/pyarrow.lib.UInt16Array.rst.txt @@ -0,0 +1,35 @@ +pyarrow\.lib\.UInt16Array +========================= + +.. currentmodule:: pyarrow.lib + +.. autoclass:: UInt16Array + + + .. automethod:: __init__ + + + .. rubric:: Methods + + .. autosummary:: + + ~UInt16Array.equals + ~UInt16Array.from_pandas + ~UInt16Array.isnull + ~UInt16Array.slice + ~UInt16Array.to_pandas + ~UInt16Array.to_pylist + ~UInt16Array.validate + + + + + + .. rubric:: Attributes + + .. autosummary:: + + ~UInt16Array.null_count + ~UInt16Array.type + + \ No newline at end of file http://git-wip-us.apache.org/repos/asf/arrow-site/blob/4d4a3202/docs/python/_sources/generated/pyarrow.lib.UInt32Array.rst.txt ---------------------------------------------------------------------- diff --git a/docs/python/_sources/generated/pyarrow.lib.UInt32Array.rst.txt b/docs/python/_sources/generated/pyarrow.lib.UInt32Array.rst.txt new file mode 100644 index 0000000..2e597e6 --- /dev/null +++ b/docs/python/_sources/generated/pyarrow.lib.UInt32Array.rst.txt @@ -0,0 +1,35 @@ +pyarrow\.lib\.UInt32Array +========================= + +.. currentmodule:: pyarrow.lib + +.. autoclass:: UInt32Array + + + .. automethod:: __init__ + + + .. rubric:: Methods + + .. autosummary:: + + ~UInt32Array.equals + ~UInt32Array.from_pandas + ~UInt32Array.isnull + ~UInt32Array.slice + ~UInt32Array.to_pandas + ~UInt32Array.to_pylist + ~UInt32Array.validate + + + + + + .. rubric:: Attributes + + .. autosummary:: + + ~UInt32Array.null_count + ~UInt32Array.type + + \ No newline at end of file http://git-wip-us.apache.org/repos/asf/arrow-site/blob/4d4a3202/docs/python/_sources/generated/pyarrow.lib.UInt64Array.rst.txt ---------------------------------------------------------------------- diff --git a/docs/python/_sources/generated/pyarrow.lib.UInt64Array.rst.txt b/docs/python/_sources/generated/pyarrow.lib.UInt64Array.rst.txt new file mode 100644 index 0000000..6e212aa --- /dev/null +++ b/docs/python/_sources/generated/pyarrow.lib.UInt64Array.rst.txt @@ -0,0 +1,35 @@ +pyarrow\.lib\.UInt64Array +========================= + +.. currentmodule:: pyarrow.lib + +.. autoclass:: UInt64Array + + + .. automethod:: __init__ + + + .. rubric:: Methods + + .. autosummary:: + + ~UInt64Array.equals + ~UInt64Array.from_pandas + ~UInt64Array.isnull + ~UInt64Array.slice + ~UInt64Array.to_pandas + ~UInt64Array.to_pylist + ~UInt64Array.validate + + + + + + .. rubric:: Attributes + + .. autosummary:: + + ~UInt64Array.null_count + ~UInt64Array.type + + \ No newline at end of file http://git-wip-us.apache.org/repos/asf/arrow-site/blob/4d4a3202/docs/python/_sources/generated/pyarrow.lib.UInt8Array.rst.txt ---------------------------------------------------------------------- diff --git a/docs/python/_sources/generated/pyarrow.lib.UInt8Array.rst.txt b/docs/python/_sources/generated/pyarrow.lib.UInt8Array.rst.txt new file mode 100644 index 0000000..cc79fc1 --- /dev/null +++ b/docs/python/_sources/generated/pyarrow.lib.UInt8Array.rst.txt @@ -0,0 +1,35 @@ +pyarrow\.lib\.UInt8Array +======================== + +.. currentmodule:: pyarrow.lib + +.. autoclass:: UInt8Array + + + .. automethod:: __init__ + + + .. rubric:: Methods + + .. autosummary:: + + ~UInt8Array.equals + ~UInt8Array.from_pandas + ~UInt8Array.isnull + ~UInt8Array.slice + ~UInt8Array.to_pandas + ~UInt8Array.to_pylist + ~UInt8Array.validate + + + + + + .. rubric:: Attributes + + .. autosummary:: + + ~UInt8Array.null_count + ~UInt8Array.type + + \ No newline at end of file http://git-wip-us.apache.org/repos/asf/arrow-site/blob/4d4a3202/docs/python/_sources/generated/pyarrow.parquet.ParquetDataset.rst.txt ---------------------------------------------------------------------- diff --git a/docs/python/_sources/generated/pyarrow.parquet.ParquetDataset.rst.txt b/docs/python/_sources/generated/pyarrow.parquet.ParquetDataset.rst.txt index 037d91d..e66f44e 100644 --- a/docs/python/_sources/generated/pyarrow.parquet.ParquetDataset.rst.txt +++ b/docs/python/_sources/generated/pyarrow.parquet.ParquetDataset.rst.txt @@ -15,6 +15,7 @@ pyarrow\.parquet\.ParquetDataset ~ParquetDataset.__init__ ~ParquetDataset.read + ~ParquetDataset.read_pandas ~ParquetDataset.validate_schemas http://git-wip-us.apache.org/repos/asf/arrow-site/blob/4d4a3202/docs/python/_sources/generated/pyarrow.parquet.ParquetFile.rst.txt ---------------------------------------------------------------------- diff --git a/docs/python/_sources/generated/pyarrow.parquet.ParquetFile.rst.txt b/docs/python/_sources/generated/pyarrow.parquet.ParquetFile.rst.txt index 3dd4e20..b35af86 100644 --- a/docs/python/_sources/generated/pyarrow.parquet.ParquetFile.rst.txt +++ b/docs/python/_sources/generated/pyarrow.parquet.ParquetFile.rst.txt @@ -15,7 +15,6 @@ pyarrow\.parquet\.ParquetFile ~ParquetFile.__init__ ~ParquetFile.read - ~ParquetFile.read_pandas ~ParquetFile.read_row_group http://git-wip-us.apache.org/repos/asf/arrow-site/blob/4d4a3202/docs/python/_sources/generated/pyarrow.parquet.read_metadata.rst.txt ---------------------------------------------------------------------- diff --git a/docs/python/_sources/generated/pyarrow.parquet.read_metadata.rst.txt b/docs/python/_sources/generated/pyarrow.parquet.read_metadata.rst.txt new file mode 100644 index 0000000..b0ef8ab --- /dev/null +++ b/docs/python/_sources/generated/pyarrow.parquet.read_metadata.rst.txt @@ -0,0 +1,6 @@ +pyarrow\.parquet\.read\_metadata +================================ + +.. currentmodule:: pyarrow.parquet + +.. autofunction:: read_metadata \ No newline at end of file http://git-wip-us.apache.org/repos/asf/arrow-site/blob/4d4a3202/docs/python/_sources/generated/pyarrow.parquet.read_pandas.rst.txt ---------------------------------------------------------------------- diff --git a/docs/python/_sources/generated/pyarrow.parquet.read_pandas.rst.txt b/docs/python/_sources/generated/pyarrow.parquet.read_pandas.rst.txt new file mode 100644 index 0000000..2bd36b7 --- /dev/null +++ b/docs/python/_sources/generated/pyarrow.parquet.read_pandas.rst.txt @@ -0,0 +1,6 @@ +pyarrow\.parquet\.read\_pandas +============================== + +.. currentmodule:: pyarrow.parquet + +.. autofunction:: read_pandas \ No newline at end of file http://git-wip-us.apache.org/repos/asf/arrow-site/blob/4d4a3202/docs/python/_sources/generated/pyarrow.parquet.read_schema.rst.txt ---------------------------------------------------------------------- diff --git a/docs/python/_sources/generated/pyarrow.parquet.read_schema.rst.txt b/docs/python/_sources/generated/pyarrow.parquet.read_schema.rst.txt new file mode 100644 index 0000000..2dce875 --- /dev/null +++ b/docs/python/_sources/generated/pyarrow.parquet.read_schema.rst.txt @@ -0,0 +1,6 @@ +pyarrow\.parquet\.read\_schema +============================== + +.. currentmodule:: pyarrow.parquet + +.. autofunction:: read_schema \ No newline at end of file http://git-wip-us.apache.org/repos/asf/arrow-site/blob/4d4a3202/docs/python/_sources/generated/pyarrow.plasma.ObjectID.rst.txt ---------------------------------------------------------------------- diff --git a/docs/python/_sources/generated/pyarrow.plasma.ObjectID.rst.txt b/docs/python/_sources/generated/pyarrow.plasma.ObjectID.rst.txt new file mode 100644 index 0000000..1979484 --- /dev/null +++ b/docs/python/_sources/generated/pyarrow.plasma.ObjectID.rst.txt @@ -0,0 +1,22 @@ +pyarrow\.plasma\.ObjectID +========================= + +.. currentmodule:: pyarrow.plasma + +.. autoclass:: ObjectID + + + .. automethod:: __init__ + + + .. rubric:: Methods + + .. autosummary:: + + ~ObjectID.binary + + + + + + \ No newline at end of file http://git-wip-us.apache.org/repos/asf/arrow-site/blob/4d4a3202/docs/python/_sources/generated/pyarrow.plasma.PlasmaBuffer.rst.txt ---------------------------------------------------------------------- diff --git a/docs/python/_sources/generated/pyarrow.plasma.PlasmaBuffer.rst.txt b/docs/python/_sources/generated/pyarrow.plasma.PlasmaBuffer.rst.txt new file mode 100644 index 0000000..a860c6a --- /dev/null +++ b/docs/python/_sources/generated/pyarrow.plasma.PlasmaBuffer.rst.txt @@ -0,0 +1,29 @@ +pyarrow\.plasma\.PlasmaBuffer +============================= + +.. currentmodule:: pyarrow.plasma + +.. autoclass:: PlasmaBuffer + + + .. automethod:: __init__ + + + .. rubric:: Methods + + .. autosummary:: + + ~PlasmaBuffer.to_pybytes + + + + + + .. rubric:: Attributes + + .. autosummary:: + + ~PlasmaBuffer.parent + ~PlasmaBuffer.size + + \ No newline at end of file http://git-wip-us.apache.org/repos/asf/arrow-site/blob/4d4a3202/docs/python/_sources/generated/pyarrow.plasma.PlasmaClient.rst.txt ---------------------------------------------------------------------- diff --git a/docs/python/_sources/generated/pyarrow.plasma.PlasmaClient.rst.txt b/docs/python/_sources/generated/pyarrow.plasma.PlasmaClient.rst.txt new file mode 100644 index 0000000..803b911 --- /dev/null +++ b/docs/python/_sources/generated/pyarrow.plasma.PlasmaClient.rst.txt @@ -0,0 +1,43 @@ +pyarrow\.plasma\.PlasmaClient +============================= + +.. currentmodule:: pyarrow.plasma + +.. autoclass:: PlasmaClient + + + .. automethod:: __init__ + + + .. rubric:: Methods + + .. autosummary:: + + ~PlasmaClient.contains + ~PlasmaClient.create + ~PlasmaClient.disconnect + ~PlasmaClient.evict + ~PlasmaClient.fetch + ~PlasmaClient.get + ~PlasmaClient.get_metadata + ~PlasmaClient.get_next_notification + ~PlasmaClient.hash + ~PlasmaClient.release + ~PlasmaClient.seal + ~PlasmaClient.subscribe + ~PlasmaClient.to_capsule + ~PlasmaClient.transfer + ~PlasmaClient.wait + + + + + + .. rubric:: Attributes + + .. autosummary:: + + ~PlasmaClient.manager_socket_name + ~PlasmaClient.store_socket_name + + \ No newline at end of file http://git-wip-us.apache.org/repos/asf/arrow-site/blob/4d4a3202/docs/python/_sources/index.rst.txt ---------------------------------------------------------------------- diff --git a/docs/python/_sources/index.rst.txt b/docs/python/_sources/index.rst.txt index a12853c..c2ae769 100644 --- a/docs/python/_sources/index.rst.txt +++ b/docs/python/_sources/index.rst.txt @@ -40,6 +40,7 @@ structures. data ipc filesystems + plasma pandas parquet api http://git-wip-us.apache.org/repos/asf/arrow-site/blob/4d4a3202/docs/python/_sources/memory.rst.txt ---------------------------------------------------------------------- diff --git a/docs/python/_sources/memory.rst.txt b/docs/python/_sources/memory.rst.txt index ccc6298..4806bbb 100644 --- a/docs/python/_sources/memory.rst.txt +++ b/docs/python/_sources/memory.rst.txt @@ -142,18 +142,6 @@ the stream is garbaged-collected, all of the memory is freed: stream = None pa.total_allocated_bytes() -Classes and functions that may allocate memory will often have an option to -pass in a custom memory pool: - -.. ipython:: python - - my_pool = pa.jemalloc_memory_pool() - my_pool - my_pool.bytes_allocated() - stream = pa.BufferOutputStream(my_pool) - stream.write(b'foo') - my_pool.bytes_allocated() - On-Disk and Memory Mapped Files ------------------------------- @@ -226,10 +214,3 @@ file interfaces that can read and write to Arrow Buffers. reader.read(7) These have similar semantics to Python's built-in ``io.BytesIO``. - -Hadoop Filesystem ------------------ - -:class:`~pyarrow.HdfsFile` is an implementation of :class:`~pyarrow.NativeFile` -that can read and write to the Hadoop filesytem. Read more in the -:ref:`Filesystems Section <hdfs>`. http://git-wip-us.apache.org/repos/asf/arrow-site/blob/4d4a3202/docs/python/_sources/parquet.rst.txt ---------------------------------------------------------------------- diff --git a/docs/python/_sources/parquet.rst.txt b/docs/python/_sources/parquet.rst.txt index 3317b99..7626c15 100644 --- a/docs/python/_sources/parquet.rst.txt +++ b/docs/python/_sources/parquet.rst.txt @@ -175,7 +175,7 @@ These settings can also be set on a per-column basis: .. code-block:: python - pa.write_table(table, where, compression={'foo': 'snappy', 'bar': 'gzip'}, + pq.write_table(table, where, compression={'foo': 'snappy', 'bar': 'gzip'}, use_dictionary=['foo', 'bar']) Reading Multiples Files and Partitioned Datasets http://git-wip-us.apache.org/repos/asf/arrow-site/blob/4d4a3202/docs/python/_sources/plasma.rst.txt ---------------------------------------------------------------------- diff --git a/docs/python/_sources/plasma.rst.txt b/docs/python/_sources/plasma.rst.txt new file mode 100644 index 0000000..832d996 --- /dev/null +++ b/docs/python/_sources/plasma.rst.txt @@ -0,0 +1,337 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + +.. currentmodule:: pyarrow +.. _plasma: + +The Plasma In-Memory Object Store +================================= + +.. contents:: Contents + :depth: 3 + + +The Plasma API +-------------- + +Starting the Plasma store +^^^^^^^^^^^^^^^^^^^^^^^^^ + +You can start the Plasma store by issuing a terminal command similar to the +following: + +.. code-block:: bash + + plasma_store -m 1000000000 -s /tmp/plasma + +The ``-m`` flag specifies the size of the store in bytes, and the ``-s`` flag +specifies the socket that the store will listen at. Thus, the above command +allows the Plasma store to use up to 1GB of memory, and sets the socket to +``/tmp/plasma``. + +Leaving the current terminal window open as long as Plasma store should keep +running. Messages, concerning such as disconnecting clients, may occasionally be +printed to the screen. To stop running the Plasma store, you can press +``Ctrl-C`` in the terminal. + +Creating a Plasma client +^^^^^^^^^^^^^^^^^^^^^^^^ + +To start a Plasma client from Python, call ``plasma.connect`` using the same +socket name: + +.. code-block:: python + + import pyarrow.plasma as plasma + client = plasma.connect("/tmp/plasma", "", 0) + +If the following error occurs from running the above Python code, that +means that either the socket given is incorrect, or the ``./plasma_store`` is +not currently running. Check to see if the Plasma store is still running. + +.. code-block:: shell + + >>> client = plasma.connect("/tmp/plasma", "", 0) + Connection to socket failed for pathname /tmp/plasma + Could not connect to socket /tmp/plasma + + +Object IDs +^^^^^^^^^^ + +Each object in the Plasma store should be associated with a unique ID. The +Object ID then serves as a key that any client can use to retrieve that object +from the Plasma store. You can form an ``ObjectID`` object from a byte string of +length 20. + +.. code-block:: shell + + # Create an ObjectID. + >>> id = plasma.ObjectID(20 * b"a") + + # The character "a" is encoded as 61 in hex. + >>> id + ObjectID(6161616161616161616161616161616161616161) + +The random generation of Object IDs is often good enough to ensure unique IDs. +You can easily create a helper function that randomly generates object IDs as +follows: + +.. code-block:: python + + import numpy as np + + def random_object_id(): + return plasma.ObjectID(np.random.bytes(20)) + + +Creating an Object +^^^^^^^^^^^^^^^^^^ + +Objects are created in Plasma in two stages. First, they are **created**, which +allocates a buffer for the object. At this point, the client can write to the +buffer and construct the object within the allocated buffer. + +To create an object for Plasma, you need to create an object ID, as well as +give the object's maximum size in bytes. + +.. code-block:: python + + # Create an object. + object_id = plasma.ObjectID(20 * b"a") + object_size = 1000 + buffer = memoryview(client.create(object_id, object_size)) + + # Write to the buffer. + for i in range(1000): + buffer[i] = i % 128 + +When the client is done, the client **seals** the buffer, making the object +immutable, and making it available to other Plasma clients. + +.. code-block:: python + + # Seal the object. This makes the object immutable and available to other clients. + client.seal(object_id) + + +Getting an Object +^^^^^^^^^^^^^^^^^ + +After an object has been sealed, any client who knows the object ID can get +the object. + +.. code-block:: python + + # Create a different client. Note that this second client could be + # created in the same or in a separate, concurrent Python session. + client2 = plasma.connect("/tmp/plasma", "", 0) + + # Get the object in the second client. This blocks until the object has been sealed. + object_id2 = plasma.ObjectID(20 * b"a") + [buffer2] = client2.get([object_id]) + +If the object has not been sealed yet, then the call to client.get will block +until the object has been sealed by the client constructing the object. Using +the ``timeout_ms`` argument to get, you can specify a timeout for this (in +milliseconds). After the timeout, the interpreter will yield control back. + +.. code-block:: shell + + >>> buffer + <memory at 0x7fdbdc96e708> + >>> buffer[1] + 1 + >>> buffer2 + <plasma.plasma.PlasmaBuffer object at 0x7fdbf2770e88> + >>> view2 = memoryview(buffer2) + >>> view2[1] + 1 + >>> view2[129] + 1 + >>> bytes(buffer[1:4]) + b'\x01\x02\x03' + >>> bytes(view2[1:4]) + b'\x01\x02\x03' + + +Using Arrow and Pandas with Plasma +---------------------------------- + +Storing Arrow Objects in Plasma +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +To store an Arrow object in Plasma, we must first **create** the object and then +**seal** it. However, Arrow objects such as ``Tensors`` may be more complicated +to write than simple binary data. + +To create the object in Plasma, you still need an ``ObjectID`` and a size to +pass in. To find out the size of your Arrow object, you can use pyarrow +API such as ``pyarrow.get_tensor_size``. + +.. code-block:: python + + import numpy as np + import pyarrow as pa + + # Create a pyarrow.Tensor object from a numpy random 2-dimensional array + data = np.random.randn(10, 4) + tensor = pa.Tensor.from_numpy(data) + + # Create the object in Plasma + object_id = plasma.ObjectID(np.random.bytes(20)) + data_size = pa.get_tensor_size(tensor) + buf = client.create(object_id, data_size) + +To write the Arrow ``Tensor`` object into the buffer, you can use Plasma to +convert the ``memoryview`` buffer into a ``pyarrow.FixedSizeBufferOutputStream`` +object. A ``pyarrow.FixedSizeBufferOutputStream`` is a format suitable for Arrow's +``pyarrow.write_tensor``: + +.. code-block:: python + + # Write the tensor into the Plasma-allocated buffer + stream = pa.FixedSizeBufferOutputStream(buf) + pa.write_tensor(tensor, stream) # Writes tensor's 552 bytes to Plasma stream + +To finish storing the Arrow object in Plasma, call ``seal``: + +.. code-block:: python + + # Seal the Plasma object + client.seal(object_id) + +Getting Arrow Objects from Plasma +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +To read the object, first retrieve it as a ``PlasmaBuffer`` using its object ID. + +.. code-block:: python + + # Get the arrow object by ObjectID. + [buf2] = client.get([object_id]) + +To convert the ``PlasmaBuffer`` back into an Arrow ``Tensor``, first create a +pyarrow ``BufferReader`` object from it. You can then pass the ``BufferReader`` +into ``pyarrow.read_tensor`` to reconstruct the Arrow ``Tensor`` object: + +.. code-block:: python + + # Reconstruct the Arrow tensor object. + reader = pa.BufferReader(buf2) + tensor2 = pa.read_tensor(reader) + +Finally, you can use ``pyarrow.read_tensor`` to convert the Arrow object +back into numpy data: + +.. code-block:: python + + # Convert back to numpy + array = tensor2.to_numpy() + +Storing Pandas DataFrames in Plasma +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Storing a Pandas ``DataFrame`` still follows the **create** then **seal** +process of storing an object in the Plasma store, however one cannot directly +write the ``DataFrame`` to Plasma with Pandas alone. Plasma also needs to know +the size of the ``DataFrame`` to allocate a buffer for. + +See :ref:`pandas` for more information on using Arrow with Pandas. + +You can create the pyarrow equivalent of a Pandas ``DataFrame`` by using +``pyarrow.from_pandas`` to convert it to a ``RecordBatch``. + +.. code-block:: python + + import pyarrow as pa + import pandas as pd + + # Create a Pandas DataFrame + d = {'one' : pd.Series([1., 2., 3.], index=['a', 'b', 'c']), + 'two' : pd.Series([1., 2., 3., 4.], index=['a', 'b', 'c', 'd'])} + df = pd.DataFrame(d) + + # Convert the Pandas DataFrame into a PyArrow RecordBatch + record_batch = pa.RecordBatch.from_pandas(df) + +Creating the Plasma object requires an ``ObjectID`` and the size of the +data. Now that we have converted the Pandas ``DataFrame`` into a PyArrow +``RecordBatch``, use the ``MockOutputStream`` to determine the +size of the Plasma object. + +.. code-block:: python + + # Create the Plasma object from the PyArrow RecordBatch. Most of the work here + # is done to determine the size of buffer to request from the object store. + object_id = plasma.ObjectID(np.random.bytes(20)) + mock_sink = pa.MockOutputStream() + stream_writer = pa.RecordBatchStreamWriter(mock_sink, record_batch.schema) + stream_writer.write_batch(record_batch) + stream_writer.close() + data_size = mock_sink.size() + buf = client.create(object_id, data_size) + +The DataFrame can now be written to the buffer as follows. + +.. code-block:: python + + # Write the PyArrow RecordBatch to Plasma + stream = pa.FixedSizeBufferOutputStream(buf) + stream_writer = pa.RecordBatchStreamWriter(stream, record_batch.schema) + stream_writer.write_batch(record_batch) + stream_writer.close() + +Finally, seal the finished object for use by all clients: + +.. code-block:: python + + # Seal the Plasma object + client.seal(object_id) + +Getting Pandas DataFrames from Plasma +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Since we store the Pandas DataFrame as a PyArrow ``RecordBatch`` object, +to get the object back from the Plasma store, we follow similar steps +to those specified in `Getting Arrow Objects from Plasma`_. + +We first have to convert the ``PlasmaBuffer`` returned from ``client.get`` +into an Arrow ``BufferReader`` object. + +.. code-block:: python + + # Fetch the Plasma object + [data] = client.get([object_id]) # Get PlasmaBuffer from ObjectID + buffer = pa.BufferReader(data) + +From the ``BufferReader``, we can create a specific ``RecordBatchStreamReader`` +in Arrow to reconstruct the stored PyArrow ``RecordBatch`` object. + +.. code-block:: python + + # Convert object back into an Arrow RecordBatch + reader = pa.RecordBatchStreamReader(buffer) + record_batch = reader.read_next_batch() + +The last step is to convert the PyArrow ``RecordBatch`` object back into +the original Pandas ``DataFrame`` structure. + +.. code-block:: python + + # Convert back into Pandas + result = record_batch.to_pandas() http://git-wip-us.apache.org/repos/asf/arrow-site/blob/4d4a3202/docs/python/_static/pygments.css ---------------------------------------------------------------------- diff --git a/docs/python/_static/pygments.css b/docs/python/_static/pygments.css index 8213e90..20c4814 100644 --- a/docs/python/_static/pygments.css +++ b/docs/python/_static/pygments.css @@ -47,8 +47,10 @@ .highlight .mh { color: #208050 } /* Literal.Number.Hex */ .highlight .mi { color: #208050 } /* Literal.Number.Integer */ .highlight .mo { color: #208050 } /* Literal.Number.Oct */ +.highlight .sa { color: #4070a0 } /* Literal.String.Affix */ .highlight .sb { color: #4070a0 } /* Literal.String.Backtick */ .highlight .sc { color: #4070a0 } /* Literal.String.Char */ +.highlight .dl { color: #4070a0 } /* Literal.String.Delimiter */ .highlight .sd { color: #4070a0; font-style: italic } /* Literal.String.Doc */ .highlight .s2 { color: #4070a0 } /* Literal.String.Double */ .highlight .se { color: #4070a0; font-weight: bold } /* Literal.String.Escape */ @@ -59,7 +61,9 @@ .highlight .s1 { color: #4070a0 } /* Literal.String.Single */ .highlight .ss { color: #517918 } /* Literal.String.Symbol */ .highlight .bp { color: #007020 } /* Name.Builtin.Pseudo */ +.highlight .fm { color: #06287e } /* Name.Function.Magic */ .highlight .vc { color: #bb60d5 } /* Name.Variable.Class */ .highlight .vg { color: #bb60d5 } /* Name.Variable.Global */ .highlight .vi { color: #bb60d5 } /* Name.Variable.Instance */ +.highlight .vm { color: #bb60d5 } /* Name.Variable.Magic */ .highlight .il { color: #208050 } /* Literal.Number.Integer.Long */ \ No newline at end of file
