http://git-wip-us.apache.org/repos/asf/arrow-site/blob/4d4a3202/docs/python/parquet.html ---------------------------------------------------------------------- diff --git a/docs/python/parquet.html b/docs/python/parquet.html index d3a08ee..47fb4f3 100644 --- a/docs/python/parquet.html +++ b/docs/python/parquet.html @@ -73,7 +73,8 @@ <li class="toctree-l1"><a class="reference internal" href="memory.html">Memory and IO Interfaces</a></li> <li class="toctree-l1"><a class="reference internal" href="data.html">In-Memory Data Model</a></li> <li class="toctree-l1"><a class="reference internal" href="ipc.html">IPC: Fast Streaming and Serialization</a></li> -<li class="toctree-l1"><a class="reference internal" href="filesystems.html">Filesystem Interfaces</a></li> +<li class="toctree-l1"><a class="reference internal" href="filesystems.html">File System Interfaces</a></li> +<li class="toctree-l1"><a class="reference internal" href="plasma.html">The Plasma In-Memory Object Store</a></li> <li class="toctree-l1"><a class="reference internal" href="pandas.html">Using PyArrow with pandas</a></li> <li class="toctree-l1 current"><a class="current reference internal" href="#">Reading and Writing the Apache Parquet Format</a></li> <li class="toctree-l1"><a class="reference internal" href="api.html">API Reference</a></li> @@ -217,7 +218,7 @@ the whole file (due to the columnar layout):</p> <span class="go">one: double</span> <span class="go">three: bool</span> <span class="go">-- metadata --</span> -<span class="go">pandas: {"pandas_version": "0.19.2", "index_columns": ["__index_level_0__"], "columns": [{"metadata": null, "numpy_type": "float64", "pandas_type": "float64", "name": "one"}, {"metadata": null, "numpy_type": "bool", "pandas_type": "bool", "name": "three"}, {"metadata": null, "numpy_type": "object", "pandas_type": "unicode", "name": "two"}, {"metadata": null, "numpy_type": "int64", "pandas_type": "int64", "name": "__index_level_0__"}]}</span> +<span class="go">pandas: {"columns": [{"name": "one", "pandas_type": "float64", "metadata": null, "numpy_type": "float64"}, {"name": "three", "pandas_type": "bool", "metadata": null, "numpy_type": "bool"}, {"name": "two", "pandas_type": "unicode", "metadata": null, "numpy_type": "object"}], "pandas_version": "0.20.3", "index_columns": ["__index_level_0__"]}</span> </pre></div> </div> <p>We need not use a string to specify the origin of the file. It can be any of:</p> @@ -237,17 +238,17 @@ maps) will perform the best.</p> <span class="gp">In [13]: </span><span class="n">parquet_file</span><span class="o">.</span><span class="n">metadata</span> <span class="gh">Out[13]: </span><span class="go"></span> -<span class="go"><pyarrow._parquet.FileMetaData object at 0x2b9643b72cc8></span> -<span class="go"> created_by: parquet-cpp version 1.1.1-SNAPSHOT</span> +<span class="go"><pyarrow._parquet.FileMetaData object at 0x7fdbd8c8ecc8></span> +<span class="go"> created_by: parquet-cpp version 1.2.1-SNAPSHOT</span> <span class="go"> num_columns: 4</span> <span class="go"> num_rows: 3</span> <span class="go"> num_row_groups: 1</span> <span class="go"> format_version: 1.0</span> -<span class="go"> serialized_size: 804</span> +<span class="go"> serialized_size: 708</span> <span class="gp">In [14]: </span><span class="n">parquet_file</span><span class="o">.</span><span class="n">schema</span> <span class="go"> Out[14]: </span> -<span class="go"><pyarrow._parquet.ParquetSchema object at 0x2b9642aecfc8></span> +<span class="go"><pyarrow._parquet.ParquetSchema object at 0x7fdbd8c9f4c8></span> <span class="go">one: DOUBLE</span> <span class="go">three: BOOLEAN</span> <span class="go">two: BYTE_ARRAY UTF8</span> @@ -269,7 +270,7 @@ concatenate them into a single table. You can read individual row groups with <span class="go">two: string</span> <span class="go">__index_level_0__: int64</span> <span class="go">-- metadata --</span> -<span class="go">pandas: {"pandas_version": "0.19.2", "index_columns": ["__index_level_0__"], "columns": [{"metadata": null, "numpy_type": "float64", "pandas_type": "float64", "name": "one"}, {"metadata": null, "numpy_type": "bool", "pandas_type": "bool", "name": "three"}, {"metadata": null, "numpy_type": "object", "pandas_type": "unicode", "name": "two"}, {"metadata": null, "numpy_type": "int64", "pandas_type": "int64", "name": "__index_level_0__"}]}</span> +<span class="go">pandas: {"columns": [{"name": "one", "pandas_type": "float64", "metadata": null, "numpy_type": "float64"}, {"name": "three", "pandas_type": "bool", "metadata": null, "numpy_type": "bool"}, {"name": "two", "pandas_type": "unicode", "metadata": null, "numpy_type": "object"}], "pandas_version": "0.20.3", "index_columns": ["__index_level_0__"]}</span> </pre></div> </div> <p>We can similarly write a Parquet file with multiple row groups by using @@ -310,7 +311,7 @@ compression by default, but Brotli, Gzip, and uncompressed are also supported:</ <p>Snappy generally results in better performance, while Gzip may yield smaller files.</p> <p>These settings can also be set on a per-column basis:</p> -<div class="highlight-python"><div class="highlight"><pre><span></span><span class="n">pa</span><span class="o">.</span><span class="n">write_table</span><span class="p">(</span><span class="n">table</span><span class="p">,</span> <span class="n">where</span><span class="p">,</span> <span class="n">compression</span><span class="o">=</span><span class="p">{</span><span class="s1">'foo'</span><span class="p">:</span> <span class="s1">'snappy'</span><span class="p">,</span> <span class="s1">'bar'</span><span class="p">:</span> <span class="s1">'gzip'</span><span class="p">},</span> +<div class="highlight-python"><div class="highlight"><pre><span></span><span class="n">pq</span><span class="o">.</span><span class="n">write_table</span><span class="p">(</span><span class="n">table</span><span class="p">,</span> <span class="n">where</span><span class="p">,</span> <span class="n">compression</span><span class="o">=</span><span class="p">{</span><span class="s1">'foo'</span><span class="p">:</span> <span class="s1">'snappy'</span><span class="p">,</span> <span class="s1">'bar'</span><span class="p">:</span> <span class="s1">'gzip'</span><span class="p">},</span> <span class="n">use_dictionary</span><span class="o">=</span><span class="p">[</span><span class="s1">'foo'</span><span class="p">,</span> <span class="s1">'bar'</span><span class="p">])</span> </pre></div> </div>
http://git-wip-us.apache.org/repos/asf/arrow-site/blob/4d4a3202/docs/python/plasma.html ---------------------------------------------------------------------- diff --git a/docs/python/plasma.html b/docs/python/plasma.html new file mode 100644 index 0000000..9c86359 --- /dev/null +++ b/docs/python/plasma.html @@ -0,0 +1,461 @@ +<!DOCTYPE html> + + +<html xmlns="http://www.w3.org/1999/xhtml";> + <head> + <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> + + <title>The Plasma In-Memory Object Store — pyarrow documentation</title> + + <link rel="stylesheet" href="_static/bootstrap-sphinx.css" type="text/css" /> + <link rel="stylesheet" href="_static/pygments.css" type="text/css" /> + + <script type="text/javascript"> + var DOCUMENTATION_OPTIONS = { + URL_ROOT: './', + VERSION: '', + COLLAPSE_INDEX: false, + FILE_SUFFIX: '.html', + HAS_SOURCE: true, + SOURCELINK_SUFFIX: '.txt' + }; + </script> + <script type="text/javascript" src="_static/jquery.js"></script> + <script type="text/javascript" src="_static/underscore.js"></script> + <script type="text/javascript" src="_static/doctools.js"></script> + <script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.0/MathJax.js?config=TeX-AMS-MML_HTMLorMML";></script> + <script type="text/javascript" src="_static/js/jquery-1.11.0.min.js"></script> + <script type="text/javascript" src="_static/js/jquery-fix.js"></script> + <script type="text/javascript" src="_static/bootstrap-3.3.6/js/bootstrap.min.js"></script> + <script type="text/javascript" src="_static/bootstrap-sphinx.js"></script> + <link rel="index" title="Index" href="genindex.html" /> + <link rel="search" title="Search" href="search.html" /> + <link rel="next" title="Using PyArrow with pandas" href="pandas.html" /> + <link rel="prev" title="pyarrow.HdfsFile" href="generated/pyarrow.HdfsFile.html" /> +<meta charset='utf-8'> +<meta http-equiv='X-UA-Compatible' content='IE=edge,chrome=1'> +<meta name='viewport' content='width=device-width, initial-scale=1.0, maximum-scale=1'> +<meta name="apple-mobile-web-app-capable" content="yes"> + + </head> + <body role="document"> + + <div id="navbar" class="navbar navbar-default navbar-fixed-top"> + <div class="container"> + <div class="navbar-header"> + <!-- .btn-navbar is used as the toggle for collapsed navbar content --> + <button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".nav-collapse"> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + </button> + <a class="navbar-brand" href="index.html"> + pyarrow</a> + <span class="navbar-text navbar-version pull-left"><b></b></span> + </div> + + <div class="collapse navbar-collapse nav-collapse"> + <ul class="nav navbar-nav"> + + + <li class="dropdown globaltoc-container"> + <a role="button" + id="dLabelGlobalToc" + data-toggle="dropdown" + data-target="#" + href="index.html">Site <b class="caret"></b></a> + <ul class="dropdown-menu globaltoc" + role="menu" + aria-labelledby="dLabelGlobalToc"><p class="caption"><span class="caption-text">Getting Started</span></p> +<ul class="current"> +<li class="toctree-l1"><a class="reference internal" href="install.html">Install PyArrow</a></li> +<li class="toctree-l1"><a class="reference internal" href="development.html">Development</a></li> +<li class="toctree-l1"><a class="reference internal" href="memory.html">Memory and IO Interfaces</a></li> +<li class="toctree-l1"><a class="reference internal" href="data.html">In-Memory Data Model</a></li> +<li class="toctree-l1"><a class="reference internal" href="ipc.html">IPC: Fast Streaming and Serialization</a></li> +<li class="toctree-l1"><a class="reference internal" href="filesystems.html">File System Interfaces</a></li> +<li class="toctree-l1 current"><a class="current reference internal" href="#">The Plasma In-Memory Object Store</a></li> +<li class="toctree-l1"><a class="reference internal" href="pandas.html">Using PyArrow with pandas</a></li> +<li class="toctree-l1"><a class="reference internal" href="parquet.html">Reading and Writing the Apache Parquet Format</a></li> +<li class="toctree-l1"><a class="reference internal" href="api.html">API Reference</a></li> +<li class="toctree-l1"><a class="reference internal" href="getting_involved.html">Getting Involved</a></li> +</ul> +</ul> +</li> + + <li class="dropdown"> + <a role="button" + id="dLabelLocalToc" + data-toggle="dropdown" + data-target="#" + href="#">Page <b class="caret"></b></a> + <ul class="dropdown-menu localtoc" + role="menu" + aria-labelledby="dLabelLocalToc"><ul> +<li><a class="reference internal" href="#">The Plasma In-Memory Object Store</a><ul> +<li><a class="reference internal" href="#the-plasma-api">The Plasma API</a><ul> +<li><a class="reference internal" href="#starting-the-plasma-store">Starting the Plasma store</a></li> +<li><a class="reference internal" href="#creating-a-plasma-client">Creating a Plasma client</a></li> +<li><a class="reference internal" href="#object-ids">Object IDs</a></li> +<li><a class="reference internal" href="#creating-an-object">Creating an Object</a></li> +<li><a class="reference internal" href="#getting-an-object">Getting an Object</a></li> +</ul> +</li> +<li><a class="reference internal" href="#using-arrow-and-pandas-with-plasma">Using Arrow and Pandas with Plasma</a><ul> +<li><a class="reference internal" href="#storing-arrow-objects-in-plasma">Storing Arrow Objects in Plasma</a></li> +<li><a class="reference internal" href="#getting-arrow-objects-from-plasma">Getting Arrow Objects from Plasma</a></li> +<li><a class="reference internal" href="#storing-pandas-dataframes-in-plasma">Storing Pandas DataFrames in Plasma</a></li> +<li><a class="reference internal" href="#getting-pandas-dataframes-from-plasma">Getting Pandas DataFrames from Plasma</a></li> +</ul> +</li> +</ul> +</li> +</ul> +</ul> +</li> + + + + + + <li> + <a href="generated/pyarrow.HdfsFile.html" title="Previous Chapter: pyarrow.HdfsFile"><span class="glyphicon glyphicon-chevron-left visible-sm"></span><span class="hidden-sm hidden-tablet">« pyarrow.HdfsFile</span> + </a> + </li> + <li> + <a href="pandas.html" title="Next Chapter: Using PyArrow with pandas"><span class="glyphicon glyphicon-chevron-right visible-sm"></span><span class="hidden-sm hidden-tablet">Using PyArrow... »</span> + </a> + </li> + + + + + + <li class="hidden-sm"> +<div id="sourcelink"> + <a href="_sources/plasma.rst.txt" + rel="nofollow">Source</a> +</div></li> + + </ul> + + + +<form class="navbar-form navbar-right" action="search.html" method="get"> + <div class="form-group"> + <input type="text" name="q" class="form-control" placeholder="Search" /> + </div> + <input type="hidden" name="check_keywords" value="yes" /> + <input type="hidden" name="area" value="default" /> +</form> + + </div> + </div> + </div> + +<div class="container"> + <div class="row"> + <div class="col-md-12 content"> + + <div class="section" id="the-plasma-in-memory-object-store"> +<span id="plasma"></span><h1><a class="toc-backref" href="#id1">The Plasma In-Memory Object Store</a><a class="headerlink" href="#the-plasma-in-memory-object-store" title="Permalink to this headline">¶</a></h1> +<div class="contents topic" id="contents"> +<p class="topic-title first">Contents</p> +<ul class="simple"> +<li><a class="reference internal" href="#the-plasma-in-memory-object-store" id="id1">The Plasma In-Memory Object Store</a><ul> +<li><a class="reference internal" href="#the-plasma-api" id="id2">The Plasma API</a><ul> +<li><a class="reference internal" href="#starting-the-plasma-store" id="id3">Starting the Plasma store</a></li> +<li><a class="reference internal" href="#creating-a-plasma-client" id="id4">Creating a Plasma client</a></li> +<li><a class="reference internal" href="#object-ids" id="id5">Object IDs</a></li> +<li><a class="reference internal" href="#creating-an-object" id="id6">Creating an Object</a></li> +<li><a class="reference internal" href="#getting-an-object" id="id7">Getting an Object</a></li> +</ul> +</li> +<li><a class="reference internal" href="#using-arrow-and-pandas-with-plasma" id="id8">Using Arrow and Pandas with Plasma</a><ul> +<li><a class="reference internal" href="#storing-arrow-objects-in-plasma" id="id9">Storing Arrow Objects in Plasma</a></li> +<li><a class="reference internal" href="#getting-arrow-objects-from-plasma" id="id10">Getting Arrow Objects from Plasma</a></li> +<li><a class="reference internal" href="#storing-pandas-dataframes-in-plasma" id="id11">Storing Pandas DataFrames in Plasma</a></li> +<li><a class="reference internal" href="#getting-pandas-dataframes-from-plasma" id="id12">Getting Pandas DataFrames from Plasma</a></li> +</ul> +</li> +</ul> +</li> +</ul> +</div> +<div class="section" id="the-plasma-api"> +<h2><a class="toc-backref" href="#id2">The Plasma API</a><a class="headerlink" href="#the-plasma-api" title="Permalink to this headline">¶</a></h2> +<div class="section" id="starting-the-plasma-store"> +<h3><a class="toc-backref" href="#id3">Starting the Plasma store</a><a class="headerlink" href="#starting-the-plasma-store" title="Permalink to this headline">¶</a></h3> +<p>You can start the Plasma store by issuing a terminal command similar to the +following:</p> +<div class="highlight-bash"><div class="highlight"><pre><span></span>plasma_store -m <span class="m">1000000000</span> -s /tmp/plasma +</pre></div> +</div> +<p>The <code class="docutils literal"><span class="pre">-m</span></code> flag specifies the size of the store in bytes, and the <code class="docutils literal"><span class="pre">-s</span></code> flag +specifies the socket that the store will listen at. Thus, the above command +allows the Plasma store to use up to 1GB of memory, and sets the socket to +<code class="docutils literal"><span class="pre">/tmp/plasma</span></code>.</p> +<p>Leaving the current terminal window open as long as Plasma store should keep +running. Messages, concerning such as disconnecting clients, may occasionally be +printed to the screen. To stop running the Plasma store, you can press +<code class="docutils literal"><span class="pre">Ctrl-C</span></code> in the terminal.</p> +</div> +<div class="section" id="creating-a-plasma-client"> +<h3><a class="toc-backref" href="#id4">Creating a Plasma client</a><a class="headerlink" href="#creating-a-plasma-client" title="Permalink to this headline">¶</a></h3> +<p>To start a Plasma client from Python, call <code class="docutils literal"><span class="pre">plasma.connect</span></code> using the same +socket name:</p> +<div class="highlight-python"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">pyarrow.plasma</span> <span class="kn">as</span> <span class="nn">plasma</span> +<span class="n">client</span> <span class="o">=</span> <span class="n">plasma</span><span class="o">.</span><span class="n">connect</span><span class="p">(</span><span class="s2">"/tmp/plasma"</span><span class="p">,</span> <span class="s2">""</span><span class="p">,</span> <span class="mi">0</span><span class="p">)</span> +</pre></div> +</div> +<p>If the following error occurs from running the above Python code, that +means that either the socket given is incorrect, or the <code class="docutils literal"><span class="pre">./plasma_store</span></code> is +not currently running. Check to see if the Plasma store is still running.</p> +<div class="highlight-shell"><div class="highlight"><pre><span></span>>>> <span class="nv">client</span> <span class="o">=</span> plasma.connect<span class="o">(</span><span class="s2">"/tmp/plasma"</span>, <span class="s2">""</span>, <span class="m">0</span><span class="o">)</span> +Connection to socket failed <span class="k">for</span> pathname /tmp/plasma +Could not connect to socket /tmp/plasma +</pre></div> +</div> +</div> +<div class="section" id="object-ids"> +<h3><a class="toc-backref" href="#id5">Object IDs</a><a class="headerlink" href="#object-ids" title="Permalink to this headline">¶</a></h3> +<p>Each object in the Plasma store should be associated with a unique ID. The +Object ID then serves as a key that any client can use to retrieve that object +from the Plasma store. You can form an <code class="docutils literal"><span class="pre">ObjectID</span></code> object from a byte string of +length 20.</p> +<div class="highlight-shell"><div class="highlight"><pre><span></span><span class="c1"># Create an ObjectID.</span> +>>> <span class="nv">id</span> <span class="o">=</span> plasma.ObjectID<span class="o">(</span><span class="m">20</span> * b<span class="s2">"a"</span><span class="o">)</span> + +<span class="c1"># The character "a" is encoded as 61 in hex.</span> +>>> id +ObjectID<span class="o">(</span><span class="m">6161616161616161616161616161616161616161</span><span class="o">)</span> +</pre></div> +</div> +<p>The random generation of Object IDs is often good enough to ensure unique IDs. +You can easily create a helper function that randomly generates object IDs as +follows:</p> +<div class="highlight-python"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">numpy</span> <span class="kn">as</span> <span class="nn">np</span> + +<span class="k">def</span> <span class="nf">random_object_id</span><span class="p">():</span> + <span class="k">return</span> <span class="n">plasma</span><span class="o">.</span><span class="n">ObjectID</span><span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">bytes</span><span class="p">(</span><span class="mi">20</span><span class="p">))</span> +</pre></div> +</div> +</div> +<div class="section" id="creating-an-object"> +<h3><a class="toc-backref" href="#id6">Creating an Object</a><a class="headerlink" href="#creating-an-object" title="Permalink to this headline">¶</a></h3> +<p>Objects are created in Plasma in two stages. First, they are <strong>created</strong>, which +allocates a buffer for the object. At this point, the client can write to the +buffer and construct the object within the allocated buffer.</p> +<p>To create an object for Plasma, you need to create an object ID, as well as +give the object’s maximum size in bytes.</p> +<div class="highlight-python"><div class="highlight"><pre><span></span><span class="c1"># Create an object.</span> +<span class="n">object_id</span> <span class="o">=</span> <span class="n">plasma</span><span class="o">.</span><span class="n">ObjectID</span><span class="p">(</span><span class="mi">20</span> <span class="o">*</span> <span class="sa">b</span><span class="s2">"a"</span><span class="p">)</span> +<span class="n">object_size</span> <span class="o">=</span> <span class="mi">1000</span> +<span class="nb">buffer</span> <span class="o">=</span> <span class="n">memoryview</span><span class="p">(</span><span class="n">client</span><span class="o">.</span><span class="n">create</span><span class="p">(</span><span class="n">object_id</span><span class="p">,</span> <span class="n">object_size</span><span class="p">))</span> + +<span class="c1"># Write to the buffer.</span> +<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">1000</span><span class="p">):</span> + <span class="nb">buffer</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="o">=</span> <span class="n">i</span> <span class="o">%</span> <span class="mi">128</span> +</pre></div> +</div> +<p>When the client is done, the client <strong>seals</strong> the buffer, making the object +immutable, and making it available to other Plasma clients.</p> +<div class="highlight-python"><div class="highlight"><pre><span></span><span class="c1"># Seal the object. This makes the object immutable and available to other clients.</span> +<span class="n">client</span><span class="o">.</span><span class="n">seal</span><span class="p">(</span><span class="n">object_id</span><span class="p">)</span> +</pre></div> +</div> +</div> +<div class="section" id="getting-an-object"> +<h3><a class="toc-backref" href="#id7">Getting an Object</a><a class="headerlink" href="#getting-an-object" title="Permalink to this headline">¶</a></h3> +<p>After an object has been sealed, any client who knows the object ID can get +the object.</p> +<div class="highlight-python"><div class="highlight"><pre><span></span><span class="c1"># Create a different client. Note that this second client could be</span> +<span class="c1"># created in the same or in a separate, concurrent Python session.</span> +<span class="n">client2</span> <span class="o">=</span> <span class="n">plasma</span><span class="o">.</span><span class="n">connect</span><span class="p">(</span><span class="s2">"/tmp/plasma"</span><span class="p">,</span> <span class="s2">""</span><span class="p">,</span> <span class="mi">0</span><span class="p">)</span> + +<span class="c1"># Get the object in the second client. This blocks until the object has been sealed.</span> +<span class="n">object_id2</span> <span class="o">=</span> <span class="n">plasma</span><span class="o">.</span><span class="n">ObjectID</span><span class="p">(</span><span class="mi">20</span> <span class="o">*</span> <span class="sa">b</span><span class="s2">"a"</span><span class="p">)</span> +<span class="p">[</span><span class="n">buffer2</span><span class="p">]</span> <span class="o">=</span> <span class="n">client2</span><span class="o">.</span><span class="n">get</span><span class="p">([</span><span class="n">object_id</span><span class="p">])</span> +</pre></div> +</div> +<p>If the object has not been sealed yet, then the call to client.get will block +until the object has been sealed by the client constructing the object. Using +the <code class="docutils literal"><span class="pre">timeout_ms</span></code> argument to get, you can specify a timeout for this (in +milliseconds). After the timeout, the interpreter will yield control back.</p> +<div class="highlight-shell"><div class="highlight"><pre><span></span>>>> buffer +<memory at 0x7fdbdc96e708> +>>> buffer<span class="o">[</span><span class="m">1</span><span class="o">]</span> +<span class="m">1</span> +>>> buffer2 +<plasma.plasma.PlasmaBuffer object at 0x7fdbf2770e88> +>>> <span class="nv">view2</span> <span class="o">=</span> memoryview<span class="o">(</span>buffer2<span class="o">)</span> +>>> view2<span class="o">[</span><span class="m">1</span><span class="o">]</span> +<span class="m">1</span> +>>> view2<span class="o">[</span><span class="m">129</span><span class="o">]</span> +<span class="m">1</span> +>>> bytes<span class="o">(</span>buffer<span class="o">[</span><span class="m">1</span>:4<span class="o">])</span> +b<span class="s1">'\x01\x02\x03'</span> +>>> bytes<span class="o">(</span>view2<span class="o">[</span><span class="m">1</span>:4<span class="o">])</span> +b<span class="s1">'\x01\x02\x03'</span> +</pre></div> +</div> +</div> +</div> +<div class="section" id="using-arrow-and-pandas-with-plasma"> +<h2><a class="toc-backref" href="#id8">Using Arrow and Pandas with Plasma</a><a class="headerlink" href="#using-arrow-and-pandas-with-plasma" title="Permalink to this headline">¶</a></h2> +<div class="section" id="storing-arrow-objects-in-plasma"> +<h3><a class="toc-backref" href="#id9">Storing Arrow Objects in Plasma</a><a class="headerlink" href="#storing-arrow-objects-in-plasma" title="Permalink to this headline">¶</a></h3> +<p>To store an Arrow object in Plasma, we must first <strong>create</strong> the object and then +<strong>seal</strong> it. However, Arrow objects such as <code class="docutils literal"><span class="pre">Tensors</span></code> may be more complicated +to write than simple binary data.</p> +<p>To create the object in Plasma, you still need an <code class="docutils literal"><span class="pre">ObjectID</span></code> and a size to +pass in. To find out the size of your Arrow object, you can use pyarrow +API such as <code class="docutils literal"><span class="pre">pyarrow.get_tensor_size</span></code>.</p> +<div class="highlight-python"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">numpy</span> <span class="kn">as</span> <span class="nn">np</span> +<span class="kn">import</span> <span class="nn">pyarrow</span> <span class="kn">as</span> <span class="nn">pa</span> + +<span class="c1"># Create a pyarrow.Tensor object from a numpy random 2-dimensional array</span> +<span class="n">data</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">randn</span><span class="p">(</span><span class="mi">10</span><span class="p">,</span> <span class="mi">4</span><span class="p">)</span> +<span class="n">tensor</span> <span class="o">=</span> <span class="n">pa</span><span class="o">.</span><span class="n">Tensor</span><span class="o">.</span><span class="n">from_numpy</span><span class="p">(</span><span class="n">data</span><span class="p">)</span> + +<span class="c1"># Create the object in Plasma</span> +<span class="n">object_id</span> <span class="o">=</span> <span class="n">plasma</span><span class="o">.</span><span class="n">ObjectID</span><span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">bytes</span><span class="p">(</span><span class="mi">20</span><span class="p">))</span> +<span class="n">data_size</span> <span class="o">=</span> <span class="n">pa</span><span class="o">.</span><span class="n">get_tensor_size</span><span class="p">(</span><span class="n">tensor</span><span class="p">)</span> +<span class="n">buf</span> <span class="o">=</span> <span class="n">client</span><span class="o">.</span><span class="n">create</span><span class="p">(</span><span class="n">object_id</span><span class="p">,</span> <span class="n">data_size</span><span class="p">)</span> +</pre></div> +</div> +<p>To write the Arrow <code class="docutils literal"><span class="pre">Tensor</span></code> object into the buffer, you can use Plasma to +convert the <code class="docutils literal"><span class="pre">memoryview</span></code> buffer into a <code class="docutils literal"><span class="pre">pyarrow.FixedSizeBufferOutputStream</span></code> +object. A <code class="docutils literal"><span class="pre">pyarrow.FixedSizeBufferOutputStream</span></code> is a format suitable for Arrow’s +<code class="docutils literal"><span class="pre">pyarrow.write_tensor</span></code>:</p> +<div class="highlight-python"><div class="highlight"><pre><span></span><span class="c1"># Write the tensor into the Plasma-allocated buffer</span> +<span class="n">stream</span> <span class="o">=</span> <span class="n">pa</span><span class="o">.</span><span class="n">FixedSizeBufferOutputStream</span><span class="p">(</span><span class="n">buf</span><span class="p">)</span> +<span class="n">pa</span><span class="o">.</span><span class="n">write_tensor</span><span class="p">(</span><span class="n">tensor</span><span class="p">,</span> <span class="n">stream</span><span class="p">)</span> <span class="c1"># Writes tensor's 552 bytes to Plasma stream</span> +</pre></div> +</div> +<p>To finish storing the Arrow object in Plasma, call <code class="docutils literal"><span class="pre">seal</span></code>:</p> +<div class="highlight-python"><div class="highlight"><pre><span></span><span class="c1"># Seal the Plasma object</span> +<span class="n">client</span><span class="o">.</span><span class="n">seal</span><span class="p">(</span><span class="n">object_id</span><span class="p">)</span> +</pre></div> +</div> +</div> +<div class="section" id="getting-arrow-objects-from-plasma"> +<h3><a class="toc-backref" href="#id10">Getting Arrow Objects from Plasma</a><a class="headerlink" href="#getting-arrow-objects-from-plasma" title="Permalink to this headline">¶</a></h3> +<p>To read the object, first retrieve it as a <code class="docutils literal"><span class="pre">PlasmaBuffer</span></code> using its object ID.</p> +<div class="highlight-python"><div class="highlight"><pre><span></span><span class="c1"># Get the arrow object by ObjectID.</span> +<span class="p">[</span><span class="n">buf2</span><span class="p">]</span> <span class="o">=</span> <span class="n">client</span><span class="o">.</span><span class="n">get</span><span class="p">([</span><span class="n">object_id</span><span class="p">])</span> +</pre></div> +</div> +<p>To convert the <code class="docutils literal"><span class="pre">PlasmaBuffer</span></code> back into an Arrow <code class="docutils literal"><span class="pre">Tensor</span></code>, first create a +pyarrow <code class="docutils literal"><span class="pre">BufferReader</span></code> object from it. You can then pass the <code class="docutils literal"><span class="pre">BufferReader</span></code> +into <code class="docutils literal"><span class="pre">pyarrow.read_tensor</span></code> to reconstruct the Arrow <code class="docutils literal"><span class="pre">Tensor</span></code> object:</p> +<div class="highlight-python"><div class="highlight"><pre><span></span><span class="c1"># Reconstruct the Arrow tensor object.</span> +<span class="n">reader</span> <span class="o">=</span> <span class="n">pa</span><span class="o">.</span><span class="n">BufferReader</span><span class="p">(</span><span class="n">buf2</span><span class="p">)</span> +<span class="n">tensor2</span> <span class="o">=</span> <span class="n">pa</span><span class="o">.</span><span class="n">read_tensor</span><span class="p">(</span><span class="n">reader</span><span class="p">)</span> +</pre></div> +</div> +<p>Finally, you can use <code class="docutils literal"><span class="pre">pyarrow.read_tensor</span></code> to convert the Arrow object +back into numpy data:</p> +<div class="highlight-python"><div class="highlight"><pre><span></span><span class="c1"># Convert back to numpy</span> +<span class="n">array</span> <span class="o">=</span> <span class="n">tensor2</span><span class="o">.</span><span class="n">to_numpy</span><span class="p">()</span> +</pre></div> +</div> +</div> +<div class="section" id="storing-pandas-dataframes-in-plasma"> +<h3><a class="toc-backref" href="#id11">Storing Pandas DataFrames in Plasma</a><a class="headerlink" href="#storing-pandas-dataframes-in-plasma" title="Permalink to this headline">¶</a></h3> +<p>Storing a Pandas <code class="docutils literal"><span class="pre">DataFrame</span></code> still follows the <strong>create</strong> then <strong>seal</strong> +process of storing an object in the Plasma store, however one cannot directly +write the <code class="docutils literal"><span class="pre">DataFrame</span></code> to Plasma with Pandas alone. Plasma also needs to know +the size of the <code class="docutils literal"><span class="pre">DataFrame</span></code> to allocate a buffer for.</p> +<p>See <a class="reference internal" href="pandas.html#pandas"><span class="std std-ref">Using PyArrow with pandas</span></a> for more information on using Arrow with Pandas.</p> +<p>You can create the pyarrow equivalent of a Pandas <code class="docutils literal"><span class="pre">DataFrame</span></code> by using +<code class="docutils literal"><span class="pre">pyarrow.from_pandas</span></code> to convert it to a <code class="docutils literal"><span class="pre">RecordBatch</span></code>.</p> +<div class="highlight-python"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">pyarrow</span> <span class="kn">as</span> <span class="nn">pa</span> +<span class="kn">import</span> <span class="nn">pandas</span> <span class="kn">as</span> <span class="nn">pd</span> + +<span class="c1"># Create a Pandas DataFrame</span> +<span class="n">d</span> <span class="o">=</span> <span class="p">{</span><span class="s1">'one'</span> <span class="p">:</span> <span class="n">pd</span><span class="o">.</span><span class="n">Series</span><span class="p">([</span><span class="mf">1.</span><span class="p">,</span> <span class="mf">2.</span><span class="p">,</span> <span class="mf">3.</span><span class="p">],</span> <span class="n">index</span><span class="o">=</span><span class="p">[</span><span class="s1">'a'</span><span class="p">,</span> <span class="s1">'b'</span><span class="p">,</span> <span class="s1">'c'</span><span class="p">]),</span> + <span class="s1">'two'</span> <span class="p">:</span> <span class="n">pd</span><span class="o">.</span><span class="n">Series</span><span class="p">([</span><span class="mf">1.</span><span class="p">,</span> <span class="mf">2.</span><span class="p">,</span> <span class="mf">3.</span><span class="p">,</span> <span class="mf">4.</span><span class="p">],</span> <span class="n">index</span><span class="o">=</span><span class="p">[</span><span class="s1">'a'</span><span class="p">,</span> <span class="s1">'b'</span><span class="p">,</span> <span class="s1">'c'</span><span class="p">,</span> <span class="s1">'d'</span><span class="p">])}</span> +<span class="n">df</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">DataFrame</span><span class="p">(</span><span class="n">d</span><span class="p">)</span> + +<span class="c1"># Convert the Pandas DataFrame into a PyArrow RecordBatch</span> +<span class="n">record_batch</span> <span class="o">=</span> <span class="n">pa</span><span class="o">.</span><span class="n">RecordBatch</span><span class="o">.</span><span class="n">from_pandas</span><span class="p">(</span><span class="n">df</span><span class="p">)</span> +</pre></div> +</div> +<p>Creating the Plasma object requires an <code class="docutils literal"><span class="pre">ObjectID</span></code> and the size of the +data. Now that we have converted the Pandas <code class="docutils literal"><span class="pre">DataFrame</span></code> into a PyArrow +<code class="docutils literal"><span class="pre">RecordBatch</span></code>, use the <code class="docutils literal"><span class="pre">MockOutputStream</span></code> to determine the +size of the Plasma object.</p> +<div class="highlight-python"><div class="highlight"><pre><span></span><span class="c1"># Create the Plasma object from the PyArrow RecordBatch. Most of the work here</span> +<span class="c1"># is done to determine the size of buffer to request from the object store.</span> +<span class="n">object_id</span> <span class="o">=</span> <span class="n">plasma</span><span class="o">.</span><span class="n">ObjectID</span><span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">bytes</span><span class="p">(</span><span class="mi">20</span><span class="p">))</span> +<span class="n">mock_sink</span> <span class="o">=</span> <span class="n">pa</span><span class="o">.</span><span class="n">MockOutputStream</span><span class="p">()</span> +<span class="n">stream_writer</span> <span class="o">=</span> <span class="n">pa</span><span class="o">.</span><span class="n">RecordBatchStreamWriter</span><span class="p">(</span><span class="n">mock_sink</span><span class="p">,</span> <span class="n">record_batch</span><span class="o">.</span><span class="n">schema</span><span class="p">)</span> +<span class="n">stream_writer</span><span class="o">.</span><span class="n">write_batch</span><span class="p">(</span><span class="n">record_batch</span><span class="p">)</span> +<span class="n">stream_writer</span><span class="o">.</span><span class="n">close</span><span class="p">()</span> +<span class="n">data_size</span> <span class="o">=</span> <span class="n">mock_sink</span><span class="o">.</span><span class="n">size</span><span class="p">()</span> +<span class="n">buf</span> <span class="o">=</span> <span class="n">client</span><span class="o">.</span><span class="n">create</span><span class="p">(</span><span class="n">object_id</span><span class="p">,</span> <span class="n">data_size</span><span class="p">)</span> +</pre></div> +</div> +<p>The DataFrame can now be written to the buffer as follows.</p> +<div class="highlight-python"><div class="highlight"><pre><span></span><span class="c1"># Write the PyArrow RecordBatch to Plasma</span> +<span class="n">stream</span> <span class="o">=</span> <span class="n">pa</span><span class="o">.</span><span class="n">FixedSizeBufferOutputStream</span><span class="p">(</span><span class="n">buf</span><span class="p">)</span> +<span class="n">stream_writer</span> <span class="o">=</span> <span class="n">pa</span><span class="o">.</span><span class="n">RecordBatchStreamWriter</span><span class="p">(</span><span class="n">stream</span><span class="p">,</span> <span class="n">record_batch</span><span class="o">.</span><span class="n">schema</span><span class="p">)</span> +<span class="n">stream_writer</span><span class="o">.</span><span class="n">write_batch</span><span class="p">(</span><span class="n">record_batch</span><span class="p">)</span> +<span class="n">stream_writer</span><span class="o">.</span><span class="n">close</span><span class="p">()</span> +</pre></div> +</div> +<p>Finally, seal the finished object for use by all clients:</p> +<div class="highlight-python"><div class="highlight"><pre><span></span><span class="c1"># Seal the Plasma object</span> +<span class="n">client</span><span class="o">.</span><span class="n">seal</span><span class="p">(</span><span class="n">object_id</span><span class="p">)</span> +</pre></div> +</div> +</div> +<div class="section" id="getting-pandas-dataframes-from-plasma"> +<h3><a class="toc-backref" href="#id12">Getting Pandas DataFrames from Plasma</a><a class="headerlink" href="#getting-pandas-dataframes-from-plasma" title="Permalink to this headline">¶</a></h3> +<p>Since we store the Pandas DataFrame as a PyArrow <code class="docutils literal"><span class="pre">RecordBatch</span></code> object, +to get the object back from the Plasma store, we follow similar steps +to those specified in <a class="reference internal" href="#getting-arrow-objects-from-plasma">Getting Arrow Objects from Plasma</a>.</p> +<p>We first have to convert the <code class="docutils literal"><span class="pre">PlasmaBuffer</span></code> returned from <code class="docutils literal"><span class="pre">client.get</span></code> +into an Arrow <code class="docutils literal"><span class="pre">BufferReader</span></code> object.</p> +<div class="highlight-python"><div class="highlight"><pre><span></span><span class="c1"># Fetch the Plasma object</span> +<span class="p">[</span><span class="n">data</span><span class="p">]</span> <span class="o">=</span> <span class="n">client</span><span class="o">.</span><span class="n">get</span><span class="p">([</span><span class="n">object_id</span><span class="p">])</span> <span class="c1"># Get PlasmaBuffer from ObjectID</span> +<span class="nb">buffer</span> <span class="o">=</span> <span class="n">pa</span><span class="o">.</span><span class="n">BufferReader</span><span class="p">(</span><span class="n">data</span><span class="p">)</span> +</pre></div> +</div> +<p>From the <code class="docutils literal"><span class="pre">BufferReader</span></code>, we can create a specific <code class="docutils literal"><span class="pre">RecordBatchStreamReader</span></code> +in Arrow to reconstruct the stored PyArrow <code class="docutils literal"><span class="pre">RecordBatch</span></code> object.</p> +<div class="highlight-python"><div class="highlight"><pre><span></span><span class="c1"># Convert object back into an Arrow RecordBatch</span> +<span class="n">reader</span> <span class="o">=</span> <span class="n">pa</span><span class="o">.</span><span class="n">RecordBatchStreamReader</span><span class="p">(</span><span class="nb">buffer</span><span class="p">)</span> +<span class="n">record_batch</span> <span class="o">=</span> <span class="n">reader</span><span class="o">.</span><span class="n">read_next_batch</span><span class="p">()</span> +</pre></div> +</div> +<p>The last step is to convert the PyArrow <code class="docutils literal"><span class="pre">RecordBatch</span></code> object back into +the original Pandas <code class="docutils literal"><span class="pre">DataFrame</span></code> structure.</p> +<div class="highlight-python"><div class="highlight"><pre><span></span><span class="c1"># Convert back into Pandas</span> +<span class="n">result</span> <span class="o">=</span> <span class="n">record_batch</span><span class="o">.</span><span class="n">to_pandas</span><span class="p">()</span> +</pre></div> +</div> +</div> +</div> +</div> + + + </div> + + </div> +</div> +<footer class="footer"> + <div class="container"> + <p class="pull-right"> + <a href="#">Back to top</a> + + </p> + <p> + © Copyright 2016-2017 Apache Software Foundation.<br/> + Created using <a href="http://sphinx-doc.org/";>Sphinx</a> 1.5.5.<br/> + </p> + </div> +</footer> + </body> +</html> \ No newline at end of file http://git-wip-us.apache.org/repos/asf/arrow-site/blob/4d4a3202/docs/python/search.html ---------------------------------------------------------------------- diff --git a/docs/python/search.html b/docs/python/search.html index cc410d7..bc8a5f8 100644 --- a/docs/python/search.html +++ b/docs/python/search.html @@ -79,7 +79,8 @@ <li class="toctree-l1"><a class="reference internal" href="memory.html">Memory and IO Interfaces</a></li> <li class="toctree-l1"><a class="reference internal" href="data.html">In-Memory Data Model</a></li> <li class="toctree-l1"><a class="reference internal" href="ipc.html">IPC: Fast Streaming and Serialization</a></li> -<li class="toctree-l1"><a class="reference internal" href="filesystems.html">Filesystem Interfaces</a></li> +<li class="toctree-l1"><a class="reference internal" href="filesystems.html">File System Interfaces</a></li> +<li class="toctree-l1"><a class="reference internal" href="plasma.html">The Plasma In-Memory Object Store</a></li> <li class="toctree-l1"><a class="reference internal" href="pandas.html">Using PyArrow with pandas</a></li> <li class="toctree-l1"><a class="reference internal" href="parquet.html">Reading and Writing the Apache Parquet Format</a></li> <li class="toctree-l1"><a class="reference internal" href="api.html">API Reference</a></li>
