wjones127 commented on a change in pull request #11913: URL: https://github.com/apache/arrow/pull/11913#discussion_r769021050
########## File path: docs/source/developers/guide/tutorials/python_tutorial.rst ########## @@ -25,3 +25,513 @@ *************** Python tutorial *************** + +In this tutorial we will make an actual feature contribution to +Arrow following the steps specified by :ref:`quick-ref-guide` +section of the guide and a more detailed :ref:`step_by_step` +section. Navigate there whenever there is some information +you may find is missing here. + +The feature contribution will be added to the compute module +in PyArrow. But you can also follow the steps in case you are +correcting a bug or adding a binding. + +This tutorial is different from the :ref:`step_by_step` as we +will be working on a specific case. This tutorial is not meant +as a step by step guide. + +**Lets start!** + +Set up +------ + +Lets setup the Arrow repository. We presume here that Git is +already installed. Otherwise please see the :ref:`set-up` section. + +Once the `Apache Arrow repository <https://github.com/apache/arrow>`_ +is forked we will clone it and add the link of the main repository +to our upstream. + +.. code:: console + + $ git clone https://github.com/<your username>/arrow.git + $ cd arrow + $ git remote add upstream https://github.com/apache/arrow + +Building PyArrow +---------------- + +Script for building PyArrow differs depending on the Operating +System you are using. For this reason we will only refer to +the instructions for the building process in this tutorial. + +.. seealso:: + + For the **introduction** to the building process refer to the + :ref:`build-arrow-guide` section. + + For the **instructions** on how to build PyArrow refer to the + :ref:`build_pyarrow` section. + +Create a JIRA issue for the new feature +--------------------------------------- + +We will add a new feature that imitates an existing function +``min_max`` from the ``arrow.compute`` module but makes the +interval bigger for 1 in both directions. Note that this is a +made-up function for the purpose of this guide. + +See the example of the ``pc.min_max`` in +`this link <https://arrow.apache.org/cookbook/py/data.html#computing-mean-min-max-values-of-an-array>`_. + +First we need to create a JIRA issue as it doesn't exist yet. +With a JIRA account created we will navigate to the +`Apache Arrow JIRA dashboard <https://issues.apache.org/jira/projects/ARROW>`_ +and click on the **Create** button. + +.. figure:: python_tutorial_jira_title.jpeg + :scale: 70 % + :alt: JIRA dashboard with a window for creating a new issue. + + Creating a JIRA issue, adding title (summary) and components. + +.. figure:: python_tutorial_jira_description.jpeg + :scale: 70 % + :alt: JIRA dashboard with a window for creating a + description for the new issue. + + Creating a JIRA issue, adding a description. + +We will also add some comments to start a conversation. + +.. figure:: python_tutorial_jira_comment.jpeg + :scale: 50 % + :alt: JIRA issue page where comment is being added. + + Adding a comment to the JIRA ticket we created. + +We have successfully created a new JIRA issue with index ARROW-14977. + +.. figure:: python_tutorial_jira_issue.jpeg + :scale: 50 % + :alt: JIRA page of the issue just created. + + Our JIRA issue. Yay! + +To see the issue in JIRA follow +`this link <https://issues.apache.org/jira/browse/ARROW-14977>`_. + +.. seealso:: + + To get more information on JIRA issues go to + :ref:`finding-issues` part of the guide. + +Start the work on a new branch +------------------------------ + +Before we start working on adding the feature we should +create a new branch from updated master. + +.. code:: console + + $ git checkout master + $ git fetch upstream + $ git pull --ff-only upstream master + $ git checkout -b ARROW-14977 + +Lets research the Arrow library to see where the ``pc.min_max`` +function is defined/connected with the C++ and get an idea +where we could implement the new feature. + +.. figure:: python_tutorial_github_search.jpeg + :scale: 50 % + :alt: Apache Arrow GitHub repository dashboard where we are + searching for a pc.min_max function reference. + + We could try to search for the function reference in a + GitHub Apache Arrow repository. + +.. figure:: python_tutorial_github_find_in_file.jpeg + :scale: 50 % + :alt: In the GitHub repository we are searching through the + test_compute.py file for the pc.min_max function. + + And search through the ``test_compute.py`` file in ``pyarrow`` + folder. + +From the search we can see that the function is tested in the +``python/pyarrow/tests/test_compute.py`` file that would mean the +function is defined in the ``compute.py`` file. + +After examining the ``compute.py`` file we can see that together +with ``_compute.py`` the functions from C++ get wrapped into Python. +We will define the new feature at the end of the ``compute.py`` file. + +Lets run some code in the Python console from ``arrow/python`` +directory in order to learn more about ``pc.min_max``. + +.. code:: console + + $ cd python + $ python + + Python 3.9.7 (default, Oct 22 2021, 13:24:00) + [Clang 13.0.0 (clang-1300.0.29.3)] on darwin + Type "help", "copyright", "credits" or "license" for more information. + +We have entered into Python console from the shell and we can +do some research: + +.. code-block:: python + + >>> import pyarrow.compute as pc + >>> data = [4, 5, 6, None, 1] + >>> data + [4, 5, 6, None, 1] + >>> pc.min_max(data) + <pyarrow.StructScalar: [('min', 1), ('max', 6)]> + >>> pc.min_max(data, skip_nulls=False) + <pyarrow.StructScalar: [('min', None), ('max', None)]> + +We will call our new feature ``pc.tutorial_min_max``. We want the +result from our function, that takes the same input data, to be +``[('min-', 0), ('max+', 7)]``. If we specify that the null value should be +included, the result should be equal to ``pc.min_max`` that is +``[('min', None), ('max', None)]``. + +Lets add the first trial code into ``arrow/python/pyarrow/compute.py`` +where we first test the call to the "min_max" function from C++: + +.. code-block:: python + + def tutorial_min_max(values, skip_nulls=True): + """ + Add docstings Review comment: ```suggestion Add docstrings ``` ########## File path: docs/source/developers/guide/tutorials/python_tutorial.rst ########## @@ -25,3 +25,513 @@ *************** Python tutorial *************** + +In this tutorial we will make an actual feature contribution to +Arrow following the steps specified by :ref:`quick-ref-guide` +section of the guide and a more detailed :ref:`step_by_step` +section. Navigate there whenever there is some information +you may find is missing here. + +The feature contribution will be added to the compute module +in PyArrow. But you can also follow the steps in case you are +correcting a bug or adding a binding. + +This tutorial is different from the :ref:`step_by_step` as we +will be working on a specific case. This tutorial is not meant +as a step by step guide. + +**Lets start!** + +Set up +------ + +Lets setup the Arrow repository. We presume here that Git is +already installed. Otherwise please see the :ref:`set-up` section. + +Once the `Apache Arrow repository <https://github.com/apache/arrow>`_ +is forked we will clone it and add the link of the main repository +to our upstream. + +.. code:: console + + $ git clone https://github.com/<your username>/arrow.git + $ cd arrow + $ git remote add upstream https://github.com/apache/arrow Review comment: One tip I have: If you initially clone apache/arrow and then rename as upstream, your local master branch will always track upstream rather than origin. (Then you can do `git pull master` instead of `git pull upstream master`.) ``` $ git clone https://github.com/apache/arrow.git $ cd arrow $ git remote rename origin upstream $ git remote add origin https://github.com/<your username>/arrow.git ``` ########## File path: docs/source/developers/guide/tutorials/python_tutorial.rst ########## @@ -25,3 +25,513 @@ *************** Python tutorial *************** + +In this tutorial we will make an actual feature contribution to +Arrow following the steps specified by :ref:`quick-ref-guide` +section of the guide and a more detailed :ref:`step_by_step` +section. Navigate there whenever there is some information +you may find is missing here. + +The feature contribution will be added to the compute module +in PyArrow. But you can also follow the steps in case you are +correcting a bug or adding a binding. + +This tutorial is different from the :ref:`step_by_step` as we +will be working on a specific case. This tutorial is not meant +as a step by step guide. + +**Lets start!** + +Set up +------ + +Lets setup the Arrow repository. We presume here that Git is +already installed. Otherwise please see the :ref:`set-up` section. + +Once the `Apache Arrow repository <https://github.com/apache/arrow>`_ +is forked we will clone it and add the link of the main repository +to our upstream. + +.. code:: console + + $ git clone https://github.com/<your username>/arrow.git + $ cd arrow + $ git remote add upstream https://github.com/apache/arrow + +Building PyArrow +---------------- + +Script for building PyArrow differs depending on the Operating +System you are using. For this reason we will only refer to +the instructions for the building process in this tutorial. + +.. seealso:: + + For the **introduction** to the building process refer to the + :ref:`build-arrow-guide` section. + + For the **instructions** on how to build PyArrow refer to the + :ref:`build_pyarrow` section. + +Create a JIRA issue for the new feature +--------------------------------------- + +We will add a new feature that imitates an existing function +``min_max`` from the ``arrow.compute`` module but makes the +interval bigger for 1 in both directions. Note that this is a +made-up function for the purpose of this guide. + +See the example of the ``pc.min_max`` in +`this link <https://arrow.apache.org/cookbook/py/data.html#computing-mean-min-max-values-of-an-array>`_. + +First we need to create a JIRA issue as it doesn't exist yet. +With a JIRA account created we will navigate to the +`Apache Arrow JIRA dashboard <https://issues.apache.org/jira/projects/ARROW>`_ +and click on the **Create** button. + +.. figure:: python_tutorial_jira_title.jpeg + :scale: 70 % + :alt: JIRA dashboard with a window for creating a new issue. + + Creating a JIRA issue, adding title (summary) and components. + +.. figure:: python_tutorial_jira_description.jpeg + :scale: 70 % + :alt: JIRA dashboard with a window for creating a + description for the new issue. + + Creating a JIRA issue, adding a description. + +We will also add some comments to start a conversation. + +.. figure:: python_tutorial_jira_comment.jpeg + :scale: 50 % + :alt: JIRA issue page where comment is being added. + + Adding a comment to the JIRA ticket we created. + +We have successfully created a new JIRA issue with index ARROW-14977. + +.. figure:: python_tutorial_jira_issue.jpeg + :scale: 50 % + :alt: JIRA page of the issue just created. + + Our JIRA issue. Yay! + +To see the issue in JIRA follow +`this link <https://issues.apache.org/jira/browse/ARROW-14977>`_. + +.. seealso:: + + To get more information on JIRA issues go to + :ref:`finding-issues` part of the guide. + +Start the work on a new branch +------------------------------ + +Before we start working on adding the feature we should +create a new branch from updated master. + +.. code:: console + + $ git checkout master + $ git fetch upstream + $ git pull --ff-only upstream master + $ git checkout -b ARROW-14977 + +Lets research the Arrow library to see where the ``pc.min_max`` +function is defined/connected with the C++ and get an idea +where we could implement the new feature. + +.. figure:: python_tutorial_github_search.jpeg + :scale: 50 % + :alt: Apache Arrow GitHub repository dashboard where we are + searching for a pc.min_max function reference. + + We could try to search for the function reference in a + GitHub Apache Arrow repository. + +.. figure:: python_tutorial_github_find_in_file.jpeg + :scale: 50 % + :alt: In the GitHub repository we are searching through the + test_compute.py file for the pc.min_max function. + + And search through the ``test_compute.py`` file in ``pyarrow`` + folder. + +From the search we can see that the function is tested in the +``python/pyarrow/tests/test_compute.py`` file that would mean the +function is defined in the ``compute.py`` file. + +After examining the ``compute.py`` file we can see that together +with ``_compute.py`` the functions from C++ get wrapped into Python. +We will define the new feature at the end of the ``compute.py`` file. + +Lets run some code in the Python console from ``arrow/python`` +directory in order to learn more about ``pc.min_max``. + +.. code:: console + + $ cd python + $ python + + Python 3.9.7 (default, Oct 22 2021, 13:24:00) + [Clang 13.0.0 (clang-1300.0.29.3)] on darwin + Type "help", "copyright", "credits" or "license" for more information. + +We have entered into Python console from the shell and we can +do some research: + +.. code-block:: python + + >>> import pyarrow.compute as pc + >>> data = [4, 5, 6, None, 1] + >>> data + [4, 5, 6, None, 1] + >>> pc.min_max(data) + <pyarrow.StructScalar: [('min', 1), ('max', 6)]> + >>> pc.min_max(data, skip_nulls=False) + <pyarrow.StructScalar: [('min', None), ('max', None)]> + +We will call our new feature ``pc.tutorial_min_max``. We want the +result from our function, that takes the same input data, to be +``[('min-', 0), ('max+', 7)]``. If we specify that the null value should be +included, the result should be equal to ``pc.min_max`` that is +``[('min', None), ('max', None)]``. + +Lets add the first trial code into ``arrow/python/pyarrow/compute.py`` +where we first test the call to the "min_max" function from C++: + +.. code-block:: python + + def tutorial_min_max(values, skip_nulls=True): + """ + Add docstings + + Parameters + ---------- + values : Array + + Returns + ------- + result : TODO + + Examples + -------- + >>> import pyarrow.compute as pc + >>> data = [4, 5, 6, None, 1] + >>> pc.tutorial_min_max(data) + <pyarrow.StructScalar: [('min-', 0), ('max+', 7)]> + """ + + options = ScalarAggregateOptions(skip_nulls=skip_nulls) + return call_function("min_max", [values], options) + +To see if this works we will need to import ``pyarrow.compute`` +again and try: + +.. code-block:: python + + >>> import pyarrow.compute as pc + >>> data = [4, 5, 6, None, 1] + >>> pc.tutorial_min_max(data) + <pyarrow.StructScalar: [('min', 1), ('max', 6)]> + +It’s working. Now we must correct the limits to get the corrected +interval. To do that we have to do some research on +``pyarrow.StructScalar``. In ``test_scalar.py`` under the +``test_struct_duplicate_fields`` we can see an example of how the +``StructScalar`` is created. We could again run the Python console Review comment: This might also be a good place for some generic advice: "In cases where we don't yet have good documentation, unit tests can be a good place to look for code examples." ########## File path: docs/source/developers/guide/tutorials/python_tutorial.rst ########## @@ -25,3 +25,513 @@ *************** Python tutorial *************** + +In this tutorial we will make an actual feature contribution to +Arrow following the steps specified by :ref:`quick-ref-guide` +section of the guide and a more detailed :ref:`step_by_step` +section. Navigate there whenever there is some information +you may find is missing here. + +The feature contribution will be added to the compute module +in PyArrow. But you can also follow the steps in case you are +correcting a bug or adding a binding. + +This tutorial is different from the :ref:`step_by_step` as we +will be working on a specific case. This tutorial is not meant +as a step by step guide. + +**Lets start!** + +Set up +------ + +Lets setup the Arrow repository. We presume here that Git is +already installed. Otherwise please see the :ref:`set-up` section. + +Once the `Apache Arrow repository <https://github.com/apache/arrow>`_ +is forked we will clone it and add the link of the main repository +to our upstream. + +.. code:: console + + $ git clone https://github.com/<your username>/arrow.git + $ cd arrow + $ git remote add upstream https://github.com/apache/arrow + +Building PyArrow +---------------- + +Script for building PyArrow differs depending on the Operating +System you are using. For this reason we will only refer to +the instructions for the building process in this tutorial. + +.. seealso:: + + For the **introduction** to the building process refer to the + :ref:`build-arrow-guide` section. + + For the **instructions** on how to build PyArrow refer to the + :ref:`build_pyarrow` section. + +Create a JIRA issue for the new feature +--------------------------------------- + +We will add a new feature that imitates an existing function +``min_max`` from the ``arrow.compute`` module but makes the +interval bigger for 1 in both directions. Note that this is a Review comment: ```suggestion interval bigger by 1 in both directions. Note that this is a ``` ########## File path: docs/source/developers/guide/tutorials/python_tutorial.rst ########## @@ -25,3 +25,513 @@ *************** Python tutorial *************** + +In this tutorial we will make an actual feature contribution to +Arrow following the steps specified by :ref:`quick-ref-guide` +section of the guide and a more detailed :ref:`step_by_step` +section. Navigate there whenever there is some information +you may find is missing here. + +The feature contribution will be added to the compute module +in PyArrow. But you can also follow the steps in case you are +correcting a bug or adding a binding. + +This tutorial is different from the :ref:`step_by_step` as we +will be working on a specific case. This tutorial is not meant +as a step by step guide. Review comment: > This tutorial is not meant as a step by step guide If I understand this right, the reason it's not a step-by-step guide is that we don't actually expect people do follow all of these steps themselves. In particular, we don't want people creating a bunch of fake Jira issues, right? ########## File path: docs/source/developers/guide/tutorials/python_tutorial.rst ########## @@ -25,3 +25,513 @@ *************** Python tutorial *************** + +In this tutorial we will make an actual feature contribution to +Arrow following the steps specified by :ref:`quick-ref-guide` +section of the guide and a more detailed :ref:`step_by_step` +section. Navigate there whenever there is some information +you may find is missing here. + +The feature contribution will be added to the compute module +in PyArrow. But you can also follow the steps in case you are +correcting a bug or adding a binding. + +This tutorial is different from the :ref:`step_by_step` as we +will be working on a specific case. This tutorial is not meant +as a step by step guide. + +**Lets start!** + +Set up +------ + +Lets setup the Arrow repository. We presume here that Git is +already installed. Otherwise please see the :ref:`set-up` section. + +Once the `Apache Arrow repository <https://github.com/apache/arrow>`_ +is forked we will clone it and add the link of the main repository +to our upstream. + +.. code:: console + + $ git clone https://github.com/<your username>/arrow.git + $ cd arrow + $ git remote add upstream https://github.com/apache/arrow + +Building PyArrow +---------------- + +Script for building PyArrow differs depending on the Operating +System you are using. For this reason we will only refer to +the instructions for the building process in this tutorial. + +.. seealso:: + + For the **introduction** to the building process refer to the + :ref:`build-arrow-guide` section. + + For the **instructions** on how to build PyArrow refer to the + :ref:`build_pyarrow` section. + +Create a JIRA issue for the new feature +--------------------------------------- + +We will add a new feature that imitates an existing function +``min_max`` from the ``arrow.compute`` module but makes the +interval bigger for 1 in both directions. Note that this is a +made-up function for the purpose of this guide. + +See the example of the ``pc.min_max`` in +`this link <https://arrow.apache.org/cookbook/py/data.html#computing-mean-min-max-values-of-an-array>`_. + +First we need to create a JIRA issue as it doesn't exist yet. +With a JIRA account created we will navigate to the +`Apache Arrow JIRA dashboard <https://issues.apache.org/jira/projects/ARROW>`_ +and click on the **Create** button. + +.. figure:: python_tutorial_jira_title.jpeg + :scale: 70 % + :alt: JIRA dashboard with a window for creating a new issue. + + Creating a JIRA issue, adding title (summary) and components. + +.. figure:: python_tutorial_jira_description.jpeg + :scale: 70 % + :alt: JIRA dashboard with a window for creating a + description for the new issue. + + Creating a JIRA issue, adding a description. + +We will also add some comments to start a conversation. + +.. figure:: python_tutorial_jira_comment.jpeg + :scale: 50 % + :alt: JIRA issue page where comment is being added. + + Adding a comment to the JIRA ticket we created. + +We have successfully created a new JIRA issue with index ARROW-14977. + +.. figure:: python_tutorial_jira_issue.jpeg + :scale: 50 % + :alt: JIRA page of the issue just created. + + Our JIRA issue. Yay! + +To see the issue in JIRA follow +`this link <https://issues.apache.org/jira/browse/ARROW-14977>`_. + +.. seealso:: + + To get more information on JIRA issues go to + :ref:`finding-issues` part of the guide. + +Start the work on a new branch +------------------------------ + +Before we start working on adding the feature we should +create a new branch from updated master. + +.. code:: console + + $ git checkout master + $ git fetch upstream + $ git pull --ff-only upstream master + $ git checkout -b ARROW-14977 + +Lets research the Arrow library to see where the ``pc.min_max`` +function is defined/connected with the C++ and get an idea +where we could implement the new feature. + +.. figure:: python_tutorial_github_search.jpeg + :scale: 50 % + :alt: Apache Arrow GitHub repository dashboard where we are + searching for a pc.min_max function reference. + + We could try to search for the function reference in a + GitHub Apache Arrow repository. + +.. figure:: python_tutorial_github_find_in_file.jpeg + :scale: 50 % + :alt: In the GitHub repository we are searching through the + test_compute.py file for the pc.min_max function. + + And search through the ``test_compute.py`` file in ``pyarrow`` + folder. + +From the search we can see that the function is tested in the +``python/pyarrow/tests/test_compute.py`` file that would mean the +function is defined in the ``compute.py`` file. + +After examining the ``compute.py`` file we can see that together +with ``_compute.py`` the functions from C++ get wrapped into Python. +We will define the new feature at the end of the ``compute.py`` file. + +Lets run some code in the Python console from ``arrow/python`` +directory in order to learn more about ``pc.min_max``. + +.. code:: console + + $ cd python + $ python + + Python 3.9.7 (default, Oct 22 2021, 13:24:00) + [Clang 13.0.0 (clang-1300.0.29.3)] on darwin + Type "help", "copyright", "credits" or "license" for more information. + +We have entered into Python console from the shell and we can +do some research: + +.. code-block:: python + + >>> import pyarrow.compute as pc + >>> data = [4, 5, 6, None, 1] + >>> data + [4, 5, 6, None, 1] + >>> pc.min_max(data) + <pyarrow.StructScalar: [('min', 1), ('max', 6)]> + >>> pc.min_max(data, skip_nulls=False) + <pyarrow.StructScalar: [('min', None), ('max', None)]> + +We will call our new feature ``pc.tutorial_min_max``. We want the +result from our function, that takes the same input data, to be +``[('min-', 0), ('max+', 7)]``. If we specify that the null value should be +included, the result should be equal to ``pc.min_max`` that is +``[('min', None), ('max', None)]``. + +Lets add the first trial code into ``arrow/python/pyarrow/compute.py`` +where we first test the call to the "min_max" function from C++: + +.. code-block:: python + + def tutorial_min_max(values, skip_nulls=True): + """ + Add docstings + + Parameters + ---------- + values : Array + + Returns + ------- + result : TODO + + Examples + -------- + >>> import pyarrow.compute as pc + >>> data = [4, 5, 6, None, 1] + >>> pc.tutorial_min_max(data) + <pyarrow.StructScalar: [('min-', 0), ('max+', 7)]> + """ + + options = ScalarAggregateOptions(skip_nulls=skip_nulls) + return call_function("min_max", [values], options) + +To see if this works we will need to import ``pyarrow.compute`` +again and try: + +.. code-block:: python + + >>> import pyarrow.compute as pc + >>> data = [4, 5, 6, None, 1] + >>> pc.tutorial_min_max(data) + <pyarrow.StructScalar: [('min', 1), ('max', 6)]> + +It’s working. Now we must correct the limits to get the corrected +interval. To do that we have to do some research on +``pyarrow.StructScalar``. In ``test_scalar.py`` under the +``test_struct_duplicate_fields`` we can see an example of how the +``StructScalar`` is created. We could again run the Python console +and try creating one ourselves. + +.. code-block:: python + + >>> import pyarrow as pa + >>> ty = pa.struct([ + ... pa.field('min-', pa.int64()), + ... pa.field('max+', pa.int64()), + ... ]) + >>> pa.scalar([('min-', 3), ('max+', 9)], type=ty) + <pyarrow.StructScalar: [('min-', 3), ('max+', 9)]> + +With the new gained knowledge about ``StructScalar`` and additional +options for the ``pc.min_max`` function we can finish the work. + +.. code-block:: python + + def tutorial_min_max(values, skip_nulls=True): + + """ + Compute the minimum-1 and maximum+1 values of a numeric array. + Null values are ignored by default. This can be changed through + ScalarAggregateOptions. Review comment: ScalarAggregationOptions seems to be internal here. Maybe delete this sentence and add the `skip_null` parameter to the `Parameters` section? ########## File path: docs/source/developers/guide/tutorials/python_tutorial.rst ########## @@ -25,3 +25,513 @@ *************** Python tutorial *************** + +In this tutorial we will make an actual feature contribution to +Arrow following the steps specified by :ref:`quick-ref-guide` +section of the guide and a more detailed :ref:`step_by_step` +section. Navigate there whenever there is some information +you may find is missing here. + +The feature contribution will be added to the compute module +in PyArrow. But you can also follow the steps in case you are +correcting a bug or adding a binding. + +This tutorial is different from the :ref:`step_by_step` as we +will be working on a specific case. This tutorial is not meant +as a step by step guide. + +**Lets start!** + +Set up +------ + +Lets setup the Arrow repository. We presume here that Git is +already installed. Otherwise please see the :ref:`set-up` section. + +Once the `Apache Arrow repository <https://github.com/apache/arrow>`_ +is forked we will clone it and add the link of the main repository +to our upstream. + +.. code:: console + + $ git clone https://github.com/<your username>/arrow.git + $ cd arrow + $ git remote add upstream https://github.com/apache/arrow + +Building PyArrow +---------------- + +Script for building PyArrow differs depending on the Operating +System you are using. For this reason we will only refer to +the instructions for the building process in this tutorial. + +.. seealso:: + + For the **introduction** to the building process refer to the + :ref:`build-arrow-guide` section. + + For the **instructions** on how to build PyArrow refer to the + :ref:`build_pyarrow` section. + +Create a JIRA issue for the new feature +--------------------------------------- + +We will add a new feature that imitates an existing function +``min_max`` from the ``arrow.compute`` module but makes the +interval bigger for 1 in both directions. Note that this is a +made-up function for the purpose of this guide. + +See the example of the ``pc.min_max`` in +`this link <https://arrow.apache.org/cookbook/py/data.html#computing-mean-min-max-values-of-an-array>`_. + +First we need to create a JIRA issue as it doesn't exist yet. +With a JIRA account created we will navigate to the +`Apache Arrow JIRA dashboard <https://issues.apache.org/jira/projects/ARROW>`_ +and click on the **Create** button. + +.. figure:: python_tutorial_jira_title.jpeg + :scale: 70 % + :alt: JIRA dashboard with a window for creating a new issue. + + Creating a JIRA issue, adding title (summary) and components. + +.. figure:: python_tutorial_jira_description.jpeg + :scale: 70 % + :alt: JIRA dashboard with a window for creating a + description for the new issue. + + Creating a JIRA issue, adding a description. + +We will also add some comments to start a conversation. + +.. figure:: python_tutorial_jira_comment.jpeg + :scale: 50 % + :alt: JIRA issue page where comment is being added. + + Adding a comment to the JIRA ticket we created. + +We have successfully created a new JIRA issue with index ARROW-14977. + +.. figure:: python_tutorial_jira_issue.jpeg + :scale: 50 % + :alt: JIRA page of the issue just created. + + Our JIRA issue. Yay! + +To see the issue in JIRA follow +`this link <https://issues.apache.org/jira/browse/ARROW-14977>`_. + +.. seealso:: + + To get more information on JIRA issues go to + :ref:`finding-issues` part of the guide. + +Start the work on a new branch +------------------------------ + +Before we start working on adding the feature we should +create a new branch from updated master. + +.. code:: console + + $ git checkout master + $ git fetch upstream + $ git pull --ff-only upstream master + $ git checkout -b ARROW-14977 + +Lets research the Arrow library to see where the ``pc.min_max`` +function is defined/connected with the C++ and get an idea +where we could implement the new feature. + +.. figure:: python_tutorial_github_search.jpeg + :scale: 50 % + :alt: Apache Arrow GitHub repository dashboard where we are + searching for a pc.min_max function reference. + + We could try to search for the function reference in a + GitHub Apache Arrow repository. + +.. figure:: python_tutorial_github_find_in_file.jpeg + :scale: 50 % + :alt: In the GitHub repository we are searching through the + test_compute.py file for the pc.min_max function. + + And search through the ``test_compute.py`` file in ``pyarrow`` + folder. + +From the search we can see that the function is tested in the +``python/pyarrow/tests/test_compute.py`` file that would mean the +function is defined in the ``compute.py`` file. + +After examining the ``compute.py`` file we can see that together +with ``_compute.py`` the functions from C++ get wrapped into Python. +We will define the new feature at the end of the ``compute.py`` file. + +Lets run some code in the Python console from ``arrow/python`` +directory in order to learn more about ``pc.min_max``. + +.. code:: console + + $ cd python + $ python + + Python 3.9.7 (default, Oct 22 2021, 13:24:00) + [Clang 13.0.0 (clang-1300.0.29.3)] on darwin + Type "help", "copyright", "credits" or "license" for more information. + +We have entered into Python console from the shell and we can +do some research: + +.. code-block:: python + + >>> import pyarrow.compute as pc + >>> data = [4, 5, 6, None, 1] + >>> data + [4, 5, 6, None, 1] + >>> pc.min_max(data) + <pyarrow.StructScalar: [('min', 1), ('max', 6)]> + >>> pc.min_max(data, skip_nulls=False) + <pyarrow.StructScalar: [('min', None), ('max', None)]> + +We will call our new feature ``pc.tutorial_min_max``. We want the +result from our function, that takes the same input data, to be +``[('min-', 0), ('max+', 7)]``. If we specify that the null value should be +included, the result should be equal to ``pc.min_max`` that is +``[('min', None), ('max', None)]``. + +Lets add the first trial code into ``arrow/python/pyarrow/compute.py`` +where we first test the call to the "min_max" function from C++: + +.. code-block:: python + + def tutorial_min_max(values, skip_nulls=True): + """ + Add docstings + + Parameters + ---------- + values : Array + + Returns + ------- + result : TODO + + Examples + -------- + >>> import pyarrow.compute as pc + >>> data = [4, 5, 6, None, 1] + >>> pc.tutorial_min_max(data) + <pyarrow.StructScalar: [('min-', 0), ('max+', 7)]> + """ + + options = ScalarAggregateOptions(skip_nulls=skip_nulls) + return call_function("min_max", [values], options) + +To see if this works we will need to import ``pyarrow.compute`` +again and try: + +.. code-block:: python + + >>> import pyarrow.compute as pc + >>> data = [4, 5, 6, None, 1] + >>> pc.tutorial_min_max(data) + <pyarrow.StructScalar: [('min', 1), ('max', 6)]> + +It’s working. Now we must correct the limits to get the corrected +interval. To do that we have to do some research on +``pyarrow.StructScalar``. In ``test_scalar.py`` under the +``test_struct_duplicate_fields`` we can see an example of how the +``StructScalar`` is created. We could again run the Python console +and try creating one ourselves. + +.. code-block:: python + + >>> import pyarrow as pa + >>> ty = pa.struct([ + ... pa.field('min-', pa.int64()), + ... pa.field('max+', pa.int64()), + ... ]) + >>> pa.scalar([('min-', 3), ('max+', 9)], type=ty) + <pyarrow.StructScalar: [('min-', 3), ('max+', 9)]> + +With the new gained knowledge about ``StructScalar`` and additional +options for the ``pc.min_max`` function we can finish the work. + +.. code-block:: python + + def tutorial_min_max(values, skip_nulls=True): + + """ + Compute the minimum-1 and maximum+1 values of a numeric array. + Null values are ignored by default. This can be changed through + ScalarAggregateOptions. + This is a made-up feature for the tutorial purposes. + Parameters + ---------- + values : Array + Returns + ------- + result : StructScalar of min-1 and max+1 + Examples + -------- + >>> import pyarrow.compute as pc + >>> data = [4, 5, 6, None, 1] + >>> pc.tutorial_min_max(data) + <pyarrow.StructScalar: [('min-', 0), ('max+', 7)]> + """ + + options = ScalarAggregateOptions(skip_nulls=skip_nulls) + min_max = call_function("min_max", [values], options) + + if min_max[0].as_py() is not None: + min_t = min_max[0].as_py()-1 + max_t = min_max[1].as_py()+1 + else: + min_t = min_max[0].as_py() + max_t = min_max[1].as_py() + + ty = pa.struct([ + pa.field('min-', pa.int64()), + pa.field('max+', pa.int64()), + ]) + return pa.scalar([('min-', min_t), ('max+', max_t)], type=ty) + +.. TODO seealso + .. For more information about the Arrow codebase visit + .. :ref:``. (link to working on the Arrow codebase section) + +Adding a test +------------- + +Now we should add a unit test to ``python/pyarrow/tests/test_compute.py`` +and run the pytest. + +.. code-block:: python + + def test_tutorial_min_max(): + arr = [4, 5, 6, None, 1] + l1 = {'min-': 0, 'max+': 7} Review comment: nit: indentation is only 1 space in function. Shouldn't that be 4? ########## File path: docs/source/developers/guide/tutorials/python_tutorial.rst ########## @@ -25,3 +25,513 @@ *************** Python tutorial *************** + +In this tutorial we will make an actual feature contribution to +Arrow following the steps specified by :ref:`quick-ref-guide` +section of the guide and a more detailed :ref:`step_by_step` +section. Navigate there whenever there is some information +you may find is missing here. + +The feature contribution will be added to the compute module +in PyArrow. But you can also follow the steps in case you are +correcting a bug or adding a binding. + +This tutorial is different from the :ref:`step_by_step` as we +will be working on a specific case. This tutorial is not meant +as a step by step guide. + +**Lets start!** + +Set up +------ + +Lets setup the Arrow repository. We presume here that Git is +already installed. Otherwise please see the :ref:`set-up` section. + +Once the `Apache Arrow repository <https://github.com/apache/arrow>`_ +is forked we will clone it and add the link of the main repository +to our upstream. + +.. code:: console + + $ git clone https://github.com/<your username>/arrow.git + $ cd arrow + $ git remote add upstream https://github.com/apache/arrow + +Building PyArrow +---------------- + +Script for building PyArrow differs depending on the Operating +System you are using. For this reason we will only refer to +the instructions for the building process in this tutorial. + +.. seealso:: + + For the **introduction** to the building process refer to the + :ref:`build-arrow-guide` section. + + For the **instructions** on how to build PyArrow refer to the + :ref:`build_pyarrow` section. + +Create a JIRA issue for the new feature +--------------------------------------- + +We will add a new feature that imitates an existing function +``min_max`` from the ``arrow.compute`` module but makes the +interval bigger for 1 in both directions. Note that this is a +made-up function for the purpose of this guide. + +See the example of the ``pc.min_max`` in +`this link <https://arrow.apache.org/cookbook/py/data.html#computing-mean-min-max-values-of-an-array>`_. + +First we need to create a JIRA issue as it doesn't exist yet. +With a JIRA account created we will navigate to the +`Apache Arrow JIRA dashboard <https://issues.apache.org/jira/projects/ARROW>`_ +and click on the **Create** button. + +.. figure:: python_tutorial_jira_title.jpeg + :scale: 70 % + :alt: JIRA dashboard with a window for creating a new issue. + + Creating a JIRA issue, adding title (summary) and components. + +.. figure:: python_tutorial_jira_description.jpeg + :scale: 70 % + :alt: JIRA dashboard with a window for creating a + description for the new issue. + + Creating a JIRA issue, adding a description. + +We will also add some comments to start a conversation. + +.. figure:: python_tutorial_jira_comment.jpeg + :scale: 50 % + :alt: JIRA issue page where comment is being added. + + Adding a comment to the JIRA ticket we created. + +We have successfully created a new JIRA issue with index ARROW-14977. + +.. figure:: python_tutorial_jira_issue.jpeg + :scale: 50 % + :alt: JIRA page of the issue just created. + + Our JIRA issue. Yay! + +To see the issue in JIRA follow +`this link <https://issues.apache.org/jira/browse/ARROW-14977>`_. + +.. seealso:: + + To get more information on JIRA issues go to + :ref:`finding-issues` part of the guide. + +Start the work on a new branch +------------------------------ + +Before we start working on adding the feature we should +create a new branch from updated master. + +.. code:: console + + $ git checkout master + $ git fetch upstream + $ git pull --ff-only upstream master + $ git checkout -b ARROW-14977 + +Lets research the Arrow library to see where the ``pc.min_max`` +function is defined/connected with the C++ and get an idea +where we could implement the new feature. + +.. figure:: python_tutorial_github_search.jpeg + :scale: 50 % + :alt: Apache Arrow GitHub repository dashboard where we are + searching for a pc.min_max function reference. + + We could try to search for the function reference in a + GitHub Apache Arrow repository. + +.. figure:: python_tutorial_github_find_in_file.jpeg + :scale: 50 % + :alt: In the GitHub repository we are searching through the + test_compute.py file for the pc.min_max function. + + And search through the ``test_compute.py`` file in ``pyarrow`` + folder. + +From the search we can see that the function is tested in the +``python/pyarrow/tests/test_compute.py`` file that would mean the +function is defined in the ``compute.py`` file. + +After examining the ``compute.py`` file we can see that together +with ``_compute.py`` the functions from C++ get wrapped into Python. +We will define the new feature at the end of the ``compute.py`` file. + +Lets run some code in the Python console from ``arrow/python`` +directory in order to learn more about ``pc.min_max``. + +.. code:: console + + $ cd python + $ python + + Python 3.9.7 (default, Oct 22 2021, 13:24:00) + [Clang 13.0.0 (clang-1300.0.29.3)] on darwin + Type "help", "copyright", "credits" or "license" for more information. + +We have entered into Python console from the shell and we can +do some research: + +.. code-block:: python + + >>> import pyarrow.compute as pc + >>> data = [4, 5, 6, None, 1] + >>> data + [4, 5, 6, None, 1] + >>> pc.min_max(data) + <pyarrow.StructScalar: [('min', 1), ('max', 6)]> + >>> pc.min_max(data, skip_nulls=False) + <pyarrow.StructScalar: [('min', None), ('max', None)]> + +We will call our new feature ``pc.tutorial_min_max``. We want the +result from our function, that takes the same input data, to be +``[('min-', 0), ('max+', 7)]``. If we specify that the null value should be +included, the result should be equal to ``pc.min_max`` that is +``[('min', None), ('max', None)]``. + +Lets add the first trial code into ``arrow/python/pyarrow/compute.py`` +where we first test the call to the "min_max" function from C++: + +.. code-block:: python + + def tutorial_min_max(values, skip_nulls=True): + """ + Add docstings + + Parameters + ---------- + values : Array + + Returns + ------- + result : TODO + + Examples + -------- + >>> import pyarrow.compute as pc + >>> data = [4, 5, 6, None, 1] + >>> pc.tutorial_min_max(data) + <pyarrow.StructScalar: [('min-', 0), ('max+', 7)]> + """ + + options = ScalarAggregateOptions(skip_nulls=skip_nulls) + return call_function("min_max", [values], options) + +To see if this works we will need to import ``pyarrow.compute`` +again and try: + +.. code-block:: python + + >>> import pyarrow.compute as pc + >>> data = [4, 5, 6, None, 1] + >>> pc.tutorial_min_max(data) + <pyarrow.StructScalar: [('min', 1), ('max', 6)]> + +It’s working. Now we must correct the limits to get the corrected +interval. To do that we have to do some research on +``pyarrow.StructScalar``. In ``test_scalar.py`` under the +``test_struct_duplicate_fields`` we can see an example of how the +``StructScalar`` is created. We could again run the Python console +and try creating one ourselves. + +.. code-block:: python + + >>> import pyarrow as pa + >>> ty = pa.struct([ + ... pa.field('min-', pa.int64()), + ... pa.field('max+', pa.int64()), + ... ]) + >>> pa.scalar([('min-', 3), ('max+', 9)], type=ty) + <pyarrow.StructScalar: [('min-', 3), ('max+', 9)]> + +With the new gained knowledge about ``StructScalar`` and additional +options for the ``pc.min_max`` function we can finish the work. + +.. code-block:: python + + def tutorial_min_max(values, skip_nulls=True): + + """ + Compute the minimum-1 and maximum+1 values of a numeric array. + Null values are ignored by default. This can be changed through + ScalarAggregateOptions. + This is a made-up feature for the tutorial purposes. + Parameters + ---------- + values : Array + Returns + ------- + result : StructScalar of min-1 and max+1 + Examples + -------- + >>> import pyarrow.compute as pc + >>> data = [4, 5, 6, None, 1] + >>> pc.tutorial_min_max(data) + <pyarrow.StructScalar: [('min-', 0), ('max+', 7)]> + """ + + options = ScalarAggregateOptions(skip_nulls=skip_nulls) + min_max = call_function("min_max", [values], options) + + if min_max[0].as_py() is not None: + min_t = min_max[0].as_py()-1 + max_t = min_max[1].as_py()+1 + else: + min_t = min_max[0].as_py() + max_t = min_max[1].as_py() + + ty = pa.struct([ + pa.field('min-', pa.int64()), + pa.field('max+', pa.int64()), + ]) + return pa.scalar([('min-', min_t), ('max+', max_t)], type=ty) + +.. TODO seealso + .. For more information about the Arrow codebase visit + .. :ref:``. (link to working on the Arrow codebase section) + +Adding a test +------------- + +Now we should add a unit test to ``python/pyarrow/tests/test_compute.py`` +and run the pytest. + +.. code-block:: python + + def test_tutorial_min_max(): + arr = [4, 5, 6, None, 1] + l1 = {'min-': 0, 'max+': 7} + l2 = {'min-': None, 'max+': None} + assert pc.tutorial_min_max(arr).as_py() == l1 + assert pc.tutorial_min_max(arr, + skip_nulls=False).as_py() == l2 + +With the unit test added we can run the pytest from the shell. + +.. code:: console + + $ cd python + $ python -m pytest pyarrow/tests/test_compute.py -k test_tutorial_min_max + ======================== test session starts ========================== + platform darwin -- Python 3.9.7, pytest-6.2.5, py-1.10.0, pluggy-1.0.0 + rootdir: /Users/alenkafrim/repos/arrow/python, configfile: setup.cfg + plugins: hypothesis-6.24.1, lazy-fixture-0.6.3 + collected 204 items / 203 deselected / 1 selected + + pyarrow/tests/test_compute.py . [100%] + + ======================== 1 passed, 203 deselected in 0.16s ============ + + + $ python -m pytest pyarrow/tests/test_compute.py + ======================== test session starts =========================== + platform darwin -- Python 3.9.7, pytest-6.2.5, py-1.10.0, pluggy-1.0.0 + rootdir: /Users/alenkafrim/repos/arrow/python, configfile: setup.cfg + plugins: hypothesis-6.24.1, lazy-fixture-0.6.3 + collected 204 items + + pyarrow/tests/test_compute.py ................................... [ 46%] + ................................................. [100%] + + ========================= 204 passed in 0.49s ========================== + +.. seealso:: + + For more information about testing see :ref:`testing` section. + +Check styling +------------- + +At the end we also need to check the styling. In Arrow we use a +utility called `Archery <https://arrow.apache.org/docs/developers/archery.html>`_ +to check if code is in line with PEP 8 style guide. + +.. code:: console + + $ archery lint --python --fix + INFO:archery:Running Python formatter (autopep8) + INFO:archery:Running Python linter (flake8) + /Users/alenkafrim/repos/arrow/python/pyarrow/tests/test_compute.py:2288:80: E501 line too long (88 > 79 characters) + +We get a warning that one line is too long. Lets +make the necessary corrections and run it again. + +.. code:: console + + $ archery lint --python --fix + INFO:archery:Running Python formatter (autopep8) + INFO:archery:Running Python linter (flake8) + +Done. Now lets make the Pull Request! + +Creating a Pull Request +----------------------- + +First lets check again the changes made to the local copy +of the Arrow library in the shell. Review comment: ```suggestion First let's review our changes to Arrow in the shell using `git diff`. ``` ########## File path: docs/source/developers/guide/tutorials/python_tutorial.rst ########## @@ -25,3 +25,513 @@ *************** Python tutorial *************** + +In this tutorial we will make an actual feature contribution to +Arrow following the steps specified by :ref:`quick-ref-guide` +section of the guide and a more detailed :ref:`step_by_step` +section. Navigate there whenever there is some information +you may find is missing here. + +The feature contribution will be added to the compute module +in PyArrow. But you can also follow the steps in case you are +correcting a bug or adding a binding. + +This tutorial is different from the :ref:`step_by_step` as we +will be working on a specific case. This tutorial is not meant +as a step by step guide. + +**Lets start!** + +Set up +------ + +Lets setup the Arrow repository. We presume here that Git is +already installed. Otherwise please see the :ref:`set-up` section. + +Once the `Apache Arrow repository <https://github.com/apache/arrow>`_ +is forked we will clone it and add the link of the main repository +to our upstream. + +.. code:: console + + $ git clone https://github.com/<your username>/arrow.git + $ cd arrow + $ git remote add upstream https://github.com/apache/arrow + +Building PyArrow +---------------- + +Script for building PyArrow differs depending on the Operating +System you are using. For this reason we will only refer to +the instructions for the building process in this tutorial. + +.. seealso:: + + For the **introduction** to the building process refer to the + :ref:`build-arrow-guide` section. + + For the **instructions** on how to build PyArrow refer to the + :ref:`build_pyarrow` section. + +Create a JIRA issue for the new feature +--------------------------------------- + +We will add a new feature that imitates an existing function +``min_max`` from the ``arrow.compute`` module but makes the +interval bigger for 1 in both directions. Note that this is a +made-up function for the purpose of this guide. + +See the example of the ``pc.min_max`` in +`this link <https://arrow.apache.org/cookbook/py/data.html#computing-mean-min-max-values-of-an-array>`_. + +First we need to create a JIRA issue as it doesn't exist yet. +With a JIRA account created we will navigate to the +`Apache Arrow JIRA dashboard <https://issues.apache.org/jira/projects/ARROW>`_ +and click on the **Create** button. + +.. figure:: python_tutorial_jira_title.jpeg + :scale: 70 % + :alt: JIRA dashboard with a window for creating a new issue. + + Creating a JIRA issue, adding title (summary) and components. + +.. figure:: python_tutorial_jira_description.jpeg + :scale: 70 % + :alt: JIRA dashboard with a window for creating a + description for the new issue. + + Creating a JIRA issue, adding a description. + +We will also add some comments to start a conversation. + +.. figure:: python_tutorial_jira_comment.jpeg + :scale: 50 % + :alt: JIRA issue page where comment is being added. + + Adding a comment to the JIRA ticket we created. + +We have successfully created a new JIRA issue with index ARROW-14977. + +.. figure:: python_tutorial_jira_issue.jpeg + :scale: 50 % + :alt: JIRA page of the issue just created. + + Our JIRA issue. Yay! + +To see the issue in JIRA follow +`this link <https://issues.apache.org/jira/browse/ARROW-14977>`_. + +.. seealso:: + + To get more information on JIRA issues go to + :ref:`finding-issues` part of the guide. + +Start the work on a new branch +------------------------------ + +Before we start working on adding the feature we should +create a new branch from updated master. + +.. code:: console + + $ git checkout master + $ git fetch upstream + $ git pull --ff-only upstream master + $ git checkout -b ARROW-14977 + +Lets research the Arrow library to see where the ``pc.min_max`` +function is defined/connected with the C++ and get an idea +where we could implement the new feature. + +.. figure:: python_tutorial_github_search.jpeg + :scale: 50 % + :alt: Apache Arrow GitHub repository dashboard where we are + searching for a pc.min_max function reference. + + We could try to search for the function reference in a + GitHub Apache Arrow repository. + +.. figure:: python_tutorial_github_find_in_file.jpeg + :scale: 50 % + :alt: In the GitHub repository we are searching through the + test_compute.py file for the pc.min_max function. + + And search through the ``test_compute.py`` file in ``pyarrow`` + folder. + +From the search we can see that the function is tested in the +``python/pyarrow/tests/test_compute.py`` file that would mean the +function is defined in the ``compute.py`` file. + +After examining the ``compute.py`` file we can see that together +with ``_compute.py`` the functions from C++ get wrapped into Python. +We will define the new feature at the end of the ``compute.py`` file. + +Lets run some code in the Python console from ``arrow/python`` +directory in order to learn more about ``pc.min_max``. + +.. code:: console + + $ cd python + $ python + + Python 3.9.7 (default, Oct 22 2021, 13:24:00) + [Clang 13.0.0 (clang-1300.0.29.3)] on darwin + Type "help", "copyright", "credits" or "license" for more information. + +We have entered into Python console from the shell and we can +do some research: + +.. code-block:: python + + >>> import pyarrow.compute as pc + >>> data = [4, 5, 6, None, 1] + >>> data + [4, 5, 6, None, 1] + >>> pc.min_max(data) + <pyarrow.StructScalar: [('min', 1), ('max', 6)]> + >>> pc.min_max(data, skip_nulls=False) + <pyarrow.StructScalar: [('min', None), ('max', None)]> + +We will call our new feature ``pc.tutorial_min_max``. We want the +result from our function, that takes the same input data, to be +``[('min-', 0), ('max+', 7)]``. If we specify that the null value should be +included, the result should be equal to ``pc.min_max`` that is +``[('min', None), ('max', None)]``. + +Lets add the first trial code into ``arrow/python/pyarrow/compute.py`` +where we first test the call to the "min_max" function from C++: + +.. code-block:: python + + def tutorial_min_max(values, skip_nulls=True): + """ + Add docstings + + Parameters + ---------- + values : Array + + Returns + ------- + result : TODO + + Examples + -------- + >>> import pyarrow.compute as pc + >>> data = [4, 5, 6, None, 1] + >>> pc.tutorial_min_max(data) + <pyarrow.StructScalar: [('min-', 0), ('max+', 7)]> + """ + + options = ScalarAggregateOptions(skip_nulls=skip_nulls) + return call_function("min_max", [values], options) + +To see if this works we will need to import ``pyarrow.compute`` +again and try: + +.. code-block:: python + + >>> import pyarrow.compute as pc + >>> data = [4, 5, 6, None, 1] + >>> pc.tutorial_min_max(data) + <pyarrow.StructScalar: [('min', 1), ('max', 6)]> + +It’s working. Now we must correct the limits to get the corrected +interval. To do that we have to do some research on +``pyarrow.StructScalar``. In ``test_scalar.py`` under the +``test_struct_duplicate_fields`` we can see an example of how the +``StructScalar`` is created. We could again run the Python console +and try creating one ourselves. + +.. code-block:: python + + >>> import pyarrow as pa + >>> ty = pa.struct([ + ... pa.field('min-', pa.int64()), + ... pa.field('max+', pa.int64()), + ... ]) + >>> pa.scalar([('min-', 3), ('max+', 9)], type=ty) + <pyarrow.StructScalar: [('min-', 3), ('max+', 9)]> + +With the new gained knowledge about ``StructScalar`` and additional +options for the ``pc.min_max`` function we can finish the work. + +.. code-block:: python + + def tutorial_min_max(values, skip_nulls=True): + + """ + Compute the minimum-1 and maximum+1 values of a numeric array. + Null values are ignored by default. This can be changed through + ScalarAggregateOptions. + This is a made-up feature for the tutorial purposes. + Parameters + ---------- + values : Array + Returns + ------- + result : StructScalar of min-1 and max+1 + Examples + -------- + >>> import pyarrow.compute as pc + >>> data = [4, 5, 6, None, 1] + >>> pc.tutorial_min_max(data) + <pyarrow.StructScalar: [('min-', 0), ('max+', 7)]> + """ + + options = ScalarAggregateOptions(skip_nulls=skip_nulls) + min_max = call_function("min_max", [values], options) + + if min_max[0].as_py() is not None: + min_t = min_max[0].as_py()-1 + max_t = min_max[1].as_py()+1 + else: + min_t = min_max[0].as_py() + max_t = min_max[1].as_py() + + ty = pa.struct([ + pa.field('min-', pa.int64()), + pa.field('max+', pa.int64()), + ]) + return pa.scalar([('min-', min_t), ('max+', max_t)], type=ty) + +.. TODO seealso + .. For more information about the Arrow codebase visit + .. :ref:``. (link to working on the Arrow codebase section) + +Adding a test +------------- + +Now we should add a unit test to ``python/pyarrow/tests/test_compute.py`` +and run the pytest. + +.. code-block:: python + + def test_tutorial_min_max(): + arr = [4, 5, 6, None, 1] + l1 = {'min-': 0, 'max+': 7} + l2 = {'min-': None, 'max+': None} + assert pc.tutorial_min_max(arr).as_py() == l1 + assert pc.tutorial_min_max(arr, + skip_nulls=False).as_py() == l2 + +With the unit test added we can run the pytest from the shell. + +.. code:: console + + $ cd python + $ python -m pytest pyarrow/tests/test_compute.py -k test_tutorial_min_max + ======================== test session starts ========================== + platform darwin -- Python 3.9.7, pytest-6.2.5, py-1.10.0, pluggy-1.0.0 + rootdir: /Users/alenkafrim/repos/arrow/python, configfile: setup.cfg + plugins: hypothesis-6.24.1, lazy-fixture-0.6.3 + collected 204 items / 203 deselected / 1 selected + + pyarrow/tests/test_compute.py . [100%] + + ======================== 1 passed, 203 deselected in 0.16s ============ + + + $ python -m pytest pyarrow/tests/test_compute.py + ======================== test session starts =========================== + platform darwin -- Python 3.9.7, pytest-6.2.5, py-1.10.0, pluggy-1.0.0 + rootdir: /Users/alenkafrim/repos/arrow/python, configfile: setup.cfg + plugins: hypothesis-6.24.1, lazy-fixture-0.6.3 + collected 204 items + + pyarrow/tests/test_compute.py ................................... [ 46%] + ................................................. [100%] + + ========================= 204 passed in 0.49s ========================== + +.. seealso:: + + For more information about testing see :ref:`testing` section. + +Check styling +------------- + +At the end we also need to check the styling. In Arrow we use a +utility called `Archery <https://arrow.apache.org/docs/developers/archery.html>`_ +to check if code is in line with PEP 8 style guide. + +.. code:: console + + $ archery lint --python --fix + INFO:archery:Running Python formatter (autopep8) + INFO:archery:Running Python linter (flake8) + /Users/alenkafrim/repos/arrow/python/pyarrow/tests/test_compute.py:2288:80: E501 line too long (88 > 79 characters) + +We get a warning that one line is too long. Lets +make the necessary corrections and run it again. + +.. code:: console + + $ archery lint --python --fix + INFO:archery:Running Python formatter (autopep8) + INFO:archery:Running Python linter (flake8) + +Done. Now lets make the Pull Request! + +Creating a Pull Request +----------------------- + +First lets check again the changes made to the local copy +of the Arrow library in the shell. Review comment: Do you also want to mention why we do this step? ########## File path: docs/source/developers/guide/tutorials/python_tutorial.rst ########## @@ -25,3 +25,513 @@ *************** Python tutorial *************** + +In this tutorial we will make an actual feature contribution to +Arrow following the steps specified by :ref:`quick-ref-guide` +section of the guide and a more detailed :ref:`step_by_step` +section. Navigate there whenever there is some information +you may find is missing here. + +The feature contribution will be added to the compute module +in PyArrow. But you can also follow the steps in case you are +correcting a bug or adding a binding. + +This tutorial is different from the :ref:`step_by_step` as we +will be working on a specific case. This tutorial is not meant +as a step by step guide. + +**Lets start!** + +Set up +------ + +Lets setup the Arrow repository. We presume here that Git is +already installed. Otherwise please see the :ref:`set-up` section. + +Once the `Apache Arrow repository <https://github.com/apache/arrow>`_ +is forked we will clone it and add the link of the main repository +to our upstream. + +.. code:: console + + $ git clone https://github.com/<your username>/arrow.git + $ cd arrow + $ git remote add upstream https://github.com/apache/arrow + +Building PyArrow +---------------- + +Script for building PyArrow differs depending on the Operating +System you are using. For this reason we will only refer to +the instructions for the building process in this tutorial. + +.. seealso:: + + For the **introduction** to the building process refer to the + :ref:`build-arrow-guide` section. + + For the **instructions** on how to build PyArrow refer to the + :ref:`build_pyarrow` section. + +Create a JIRA issue for the new feature +--------------------------------------- + +We will add a new feature that imitates an existing function +``min_max`` from the ``arrow.compute`` module but makes the +interval bigger for 1 in both directions. Note that this is a +made-up function for the purpose of this guide. + +See the example of the ``pc.min_max`` in +`this link <https://arrow.apache.org/cookbook/py/data.html#computing-mean-min-max-values-of-an-array>`_. + +First we need to create a JIRA issue as it doesn't exist yet. +With a JIRA account created we will navigate to the +`Apache Arrow JIRA dashboard <https://issues.apache.org/jira/projects/ARROW>`_ +and click on the **Create** button. + +.. figure:: python_tutorial_jira_title.jpeg + :scale: 70 % + :alt: JIRA dashboard with a window for creating a new issue. + + Creating a JIRA issue, adding title (summary) and components. + +.. figure:: python_tutorial_jira_description.jpeg + :scale: 70 % + :alt: JIRA dashboard with a window for creating a + description for the new issue. + + Creating a JIRA issue, adding a description. + +We will also add some comments to start a conversation. + +.. figure:: python_tutorial_jira_comment.jpeg + :scale: 50 % + :alt: JIRA issue page where comment is being added. + + Adding a comment to the JIRA ticket we created. + +We have successfully created a new JIRA issue with index ARROW-14977. + +.. figure:: python_tutorial_jira_issue.jpeg + :scale: 50 % + :alt: JIRA page of the issue just created. + + Our JIRA issue. Yay! + +To see the issue in JIRA follow +`this link <https://issues.apache.org/jira/browse/ARROW-14977>`_. + +.. seealso:: + + To get more information on JIRA issues go to + :ref:`finding-issues` part of the guide. + +Start the work on a new branch +------------------------------ + +Before we start working on adding the feature we should +create a new branch from updated master. + +.. code:: console + + $ git checkout master + $ git fetch upstream + $ git pull --ff-only upstream master + $ git checkout -b ARROW-14977 + +Lets research the Arrow library to see where the ``pc.min_max`` +function is defined/connected with the C++ and get an idea +where we could implement the new feature. + +.. figure:: python_tutorial_github_search.jpeg + :scale: 50 % + :alt: Apache Arrow GitHub repository dashboard where we are + searching for a pc.min_max function reference. + + We could try to search for the function reference in a + GitHub Apache Arrow repository. + +.. figure:: python_tutorial_github_find_in_file.jpeg + :scale: 50 % + :alt: In the GitHub repository we are searching through the + test_compute.py file for the pc.min_max function. + + And search through the ``test_compute.py`` file in ``pyarrow`` + folder. + +From the search we can see that the function is tested in the +``python/pyarrow/tests/test_compute.py`` file that would mean the +function is defined in the ``compute.py`` file. + +After examining the ``compute.py`` file we can see that together +with ``_compute.py`` the functions from C++ get wrapped into Python. +We will define the new feature at the end of the ``compute.py`` file. + +Lets run some code in the Python console from ``arrow/python`` +directory in order to learn more about ``pc.min_max``. + +.. code:: console + + $ cd python + $ python + + Python 3.9.7 (default, Oct 22 2021, 13:24:00) + [Clang 13.0.0 (clang-1300.0.29.3)] on darwin + Type "help", "copyright", "credits" or "license" for more information. + +We have entered into Python console from the shell and we can +do some research: + +.. code-block:: python + + >>> import pyarrow.compute as pc + >>> data = [4, 5, 6, None, 1] + >>> data + [4, 5, 6, None, 1] + >>> pc.min_max(data) + <pyarrow.StructScalar: [('min', 1), ('max', 6)]> + >>> pc.min_max(data, skip_nulls=False) + <pyarrow.StructScalar: [('min', None), ('max', None)]> + +We will call our new feature ``pc.tutorial_min_max``. We want the +result from our function, that takes the same input data, to be +``[('min-', 0), ('max+', 7)]``. If we specify that the null value should be +included, the result should be equal to ``pc.min_max`` that is +``[('min', None), ('max', None)]``. + +Lets add the first trial code into ``arrow/python/pyarrow/compute.py`` +where we first test the call to the "min_max" function from C++: + +.. code-block:: python + + def tutorial_min_max(values, skip_nulls=True): + """ + Add docstings + + Parameters + ---------- + values : Array + + Returns + ------- + result : TODO + + Examples + -------- + >>> import pyarrow.compute as pc + >>> data = [4, 5, 6, None, 1] + >>> pc.tutorial_min_max(data) + <pyarrow.StructScalar: [('min-', 0), ('max+', 7)]> + """ + + options = ScalarAggregateOptions(skip_nulls=skip_nulls) + return call_function("min_max", [values], options) + +To see if this works we will need to import ``pyarrow.compute`` +again and try: + +.. code-block:: python + + >>> import pyarrow.compute as pc + >>> data = [4, 5, 6, None, 1] + >>> pc.tutorial_min_max(data) + <pyarrow.StructScalar: [('min', 1), ('max', 6)]> + +It’s working. Now we must correct the limits to get the corrected +interval. To do that we have to do some research on +``pyarrow.StructScalar``. In ``test_scalar.py`` under the +``test_struct_duplicate_fields`` we can see an example of how the +``StructScalar`` is created. We could again run the Python console +and try creating one ourselves. + +.. code-block:: python + + >>> import pyarrow as pa + >>> ty = pa.struct([ + ... pa.field('min-', pa.int64()), + ... pa.field('max+', pa.int64()), + ... ]) + >>> pa.scalar([('min-', 3), ('max+', 9)], type=ty) + <pyarrow.StructScalar: [('min-', 3), ('max+', 9)]> + +With the new gained knowledge about ``StructScalar`` and additional +options for the ``pc.min_max`` function we can finish the work. + +.. code-block:: python + + def tutorial_min_max(values, skip_nulls=True): + + """ + Compute the minimum-1 and maximum+1 values of a numeric array. + Null values are ignored by default. This can be changed through + ScalarAggregateOptions. + This is a made-up feature for the tutorial purposes. + Parameters + ---------- + values : Array + Returns + ------- + result : StructScalar of min-1 and max+1 + Examples + -------- + >>> import pyarrow.compute as pc + >>> data = [4, 5, 6, None, 1] + >>> pc.tutorial_min_max(data) + <pyarrow.StructScalar: [('min-', 0), ('max+', 7)]> + """ + + options = ScalarAggregateOptions(skip_nulls=skip_nulls) + min_max = call_function("min_max", [values], options) + + if min_max[0].as_py() is not None: + min_t = min_max[0].as_py()-1 + max_t = min_max[1].as_py()+1 + else: + min_t = min_max[0].as_py() + max_t = min_max[1].as_py() + + ty = pa.struct([ + pa.field('min-', pa.int64()), + pa.field('max+', pa.int64()), + ]) + return pa.scalar([('min-', min_t), ('max+', max_t)], type=ty) + +.. TODO seealso + .. For more information about the Arrow codebase visit + .. :ref:``. (link to working on the Arrow codebase section) + +Adding a test +------------- + +Now we should add a unit test to ``python/pyarrow/tests/test_compute.py`` +and run the pytest. + +.. code-block:: python + + def test_tutorial_min_max(): + arr = [4, 5, 6, None, 1] + l1 = {'min-': 0, 'max+': 7} + l2 = {'min-': None, 'max+': None} + assert pc.tutorial_min_max(arr).as_py() == l1 + assert pc.tutorial_min_max(arr, + skip_nulls=False).as_py() == l2 + +With the unit test added we can run the pytest from the shell. Review comment: Nice! I didn't know about the `-k` parameter. Maybe you should write advice explicitly? Something like "To run a specific unit test, pass in the test name to the ``-k`` parameter." ########## File path: docs/source/developers/guide/tutorials/python_tutorial.rst ########## @@ -25,3 +25,513 @@ *************** Python tutorial *************** + +In this tutorial we will make an actual feature contribution to +Arrow following the steps specified by :ref:`quick-ref-guide` +section of the guide and a more detailed :ref:`step_by_step` +section. Navigate there whenever there is some information +you may find is missing here. + +The feature contribution will be added to the compute module +in PyArrow. But you can also follow the steps in case you are +correcting a bug or adding a binding. + +This tutorial is different from the :ref:`step_by_step` as we +will be working on a specific case. This tutorial is not meant +as a step by step guide. + +**Lets start!** + +Set up +------ + +Lets setup the Arrow repository. We presume here that Git is +already installed. Otherwise please see the :ref:`set-up` section. + +Once the `Apache Arrow repository <https://github.com/apache/arrow>`_ +is forked we will clone it and add the link of the main repository +to our upstream. + +.. code:: console + + $ git clone https://github.com/<your username>/arrow.git + $ cd arrow + $ git remote add upstream https://github.com/apache/arrow + +Building PyArrow +---------------- + +Script for building PyArrow differs depending on the Operating +System you are using. For this reason we will only refer to +the instructions for the building process in this tutorial. + +.. seealso:: + + For the **introduction** to the building process refer to the + :ref:`build-arrow-guide` section. + + For the **instructions** on how to build PyArrow refer to the + :ref:`build_pyarrow` section. + +Create a JIRA issue for the new feature +--------------------------------------- + +We will add a new feature that imitates an existing function +``min_max`` from the ``arrow.compute`` module but makes the +interval bigger for 1 in both directions. Note that this is a +made-up function for the purpose of this guide. + +See the example of the ``pc.min_max`` in +`this link <https://arrow.apache.org/cookbook/py/data.html#computing-mean-min-max-values-of-an-array>`_. + +First we need to create a JIRA issue as it doesn't exist yet. +With a JIRA account created we will navigate to the +`Apache Arrow JIRA dashboard <https://issues.apache.org/jira/projects/ARROW>`_ +and click on the **Create** button. + +.. figure:: python_tutorial_jira_title.jpeg + :scale: 70 % + :alt: JIRA dashboard with a window for creating a new issue. + + Creating a JIRA issue, adding title (summary) and components. + +.. figure:: python_tutorial_jira_description.jpeg + :scale: 70 % + :alt: JIRA dashboard with a window for creating a + description for the new issue. + + Creating a JIRA issue, adding a description. + +We will also add some comments to start a conversation. + +.. figure:: python_tutorial_jira_comment.jpeg + :scale: 50 % + :alt: JIRA issue page where comment is being added. + + Adding a comment to the JIRA ticket we created. + +We have successfully created a new JIRA issue with index ARROW-14977. + +.. figure:: python_tutorial_jira_issue.jpeg + :scale: 50 % + :alt: JIRA page of the issue just created. + + Our JIRA issue. Yay! + +To see the issue in JIRA follow +`this link <https://issues.apache.org/jira/browse/ARROW-14977>`_. + +.. seealso:: + + To get more information on JIRA issues go to + :ref:`finding-issues` part of the guide. + +Start the work on a new branch +------------------------------ + +Before we start working on adding the feature we should +create a new branch from updated master. + +.. code:: console + + $ git checkout master + $ git fetch upstream + $ git pull --ff-only upstream master + $ git checkout -b ARROW-14977 + +Lets research the Arrow library to see where the ``pc.min_max`` +function is defined/connected with the C++ and get an idea +where we could implement the new feature. + +.. figure:: python_tutorial_github_search.jpeg + :scale: 50 % + :alt: Apache Arrow GitHub repository dashboard where we are + searching for a pc.min_max function reference. + + We could try to search for the function reference in a + GitHub Apache Arrow repository. + +.. figure:: python_tutorial_github_find_in_file.jpeg + :scale: 50 % + :alt: In the GitHub repository we are searching through the + test_compute.py file for the pc.min_max function. + + And search through the ``test_compute.py`` file in ``pyarrow`` + folder. + +From the search we can see that the function is tested in the +``python/pyarrow/tests/test_compute.py`` file that would mean the +function is defined in the ``compute.py`` file. + +After examining the ``compute.py`` file we can see that together +with ``_compute.py`` the functions from C++ get wrapped into Python. +We will define the new feature at the end of the ``compute.py`` file. + +Lets run some code in the Python console from ``arrow/python`` +directory in order to learn more about ``pc.min_max``. + +.. code:: console + + $ cd python + $ python + + Python 3.9.7 (default, Oct 22 2021, 13:24:00) + [Clang 13.0.0 (clang-1300.0.29.3)] on darwin + Type "help", "copyright", "credits" or "license" for more information. + +We have entered into Python console from the shell and we can +do some research: + +.. code-block:: python + + >>> import pyarrow.compute as pc + >>> data = [4, 5, 6, None, 1] + >>> data + [4, 5, 6, None, 1] + >>> pc.min_max(data) + <pyarrow.StructScalar: [('min', 1), ('max', 6)]> + >>> pc.min_max(data, skip_nulls=False) + <pyarrow.StructScalar: [('min', None), ('max', None)]> + +We will call our new feature ``pc.tutorial_min_max``. We want the +result from our function, that takes the same input data, to be +``[('min-', 0), ('max+', 7)]``. If we specify that the null value should be +included, the result should be equal to ``pc.min_max`` that is +``[('min', None), ('max', None)]``. + +Lets add the first trial code into ``arrow/python/pyarrow/compute.py`` +where we first test the call to the "min_max" function from C++: + +.. code-block:: python + + def tutorial_min_max(values, skip_nulls=True): + """ + Add docstings + + Parameters + ---------- + values : Array + + Returns + ------- + result : TODO + + Examples + -------- + >>> import pyarrow.compute as pc + >>> data = [4, 5, 6, None, 1] + >>> pc.tutorial_min_max(data) + <pyarrow.StructScalar: [('min-', 0), ('max+', 7)]> + """ + + options = ScalarAggregateOptions(skip_nulls=skip_nulls) + return call_function("min_max", [values], options) + +To see if this works we will need to import ``pyarrow.compute`` +again and try: + +.. code-block:: python + + >>> import pyarrow.compute as pc + >>> data = [4, 5, 6, None, 1] + >>> pc.tutorial_min_max(data) + <pyarrow.StructScalar: [('min', 1), ('max', 6)]> + +It’s working. Now we must correct the limits to get the corrected +interval. To do that we have to do some research on +``pyarrow.StructScalar``. In ``test_scalar.py`` under the +``test_struct_duplicate_fields`` we can see an example of how the +``StructScalar`` is created. We could again run the Python console +and try creating one ourselves. + +.. code-block:: python + + >>> import pyarrow as pa + >>> ty = pa.struct([ + ... pa.field('min-', pa.int64()), + ... pa.field('max+', pa.int64()), + ... ]) + >>> pa.scalar([('min-', 3), ('max+', 9)], type=ty) + <pyarrow.StructScalar: [('min-', 3), ('max+', 9)]> + +With the new gained knowledge about ``StructScalar`` and additional +options for the ``pc.min_max`` function we can finish the work. + +.. code-block:: python + + def tutorial_min_max(values, skip_nulls=True): + + """ + Compute the minimum-1 and maximum+1 values of a numeric array. + Null values are ignored by default. This can be changed through + ScalarAggregateOptions. + This is a made-up feature for the tutorial purposes. + Parameters + ---------- + values : Array + Returns + ------- + result : StructScalar of min-1 and max+1 + Examples + -------- + >>> import pyarrow.compute as pc + >>> data = [4, 5, 6, None, 1] + >>> pc.tutorial_min_max(data) + <pyarrow.StructScalar: [('min-', 0), ('max+', 7)]> + """ + + options = ScalarAggregateOptions(skip_nulls=skip_nulls) + min_max = call_function("min_max", [values], options) + + if min_max[0].as_py() is not None: + min_t = min_max[0].as_py()-1 + max_t = min_max[1].as_py()+1 + else: + min_t = min_max[0].as_py() + max_t = min_max[1].as_py() + + ty = pa.struct([ + pa.field('min-', pa.int64()), + pa.field('max+', pa.int64()), + ]) + return pa.scalar([('min-', min_t), ('max+', max_t)], type=ty) + +.. TODO seealso + .. For more information about the Arrow codebase visit + .. :ref:``. (link to working on the Arrow codebase section) + +Adding a test +------------- + +Now we should add a unit test to ``python/pyarrow/tests/test_compute.py`` +and run the pytest. + +.. code-block:: python + + def test_tutorial_min_max(): + arr = [4, 5, 6, None, 1] + l1 = {'min-': 0, 'max+': 7} + l2 = {'min-': None, 'max+': None} + assert pc.tutorial_min_max(arr).as_py() == l1 + assert pc.tutorial_min_max(arr, + skip_nulls=False).as_py() == l2 + +With the unit test added we can run the pytest from the shell. + +.. code:: console + + $ cd python + $ python -m pytest pyarrow/tests/test_compute.py -k test_tutorial_min_max + ======================== test session starts ========================== + platform darwin -- Python 3.9.7, pytest-6.2.5, py-1.10.0, pluggy-1.0.0 + rootdir: /Users/alenkafrim/repos/arrow/python, configfile: setup.cfg + plugins: hypothesis-6.24.1, lazy-fixture-0.6.3 + collected 204 items / 203 deselected / 1 selected + + pyarrow/tests/test_compute.py . [100%] + + ======================== 1 passed, 203 deselected in 0.16s ============ + + + $ python -m pytest pyarrow/tests/test_compute.py + ======================== test session starts =========================== + platform darwin -- Python 3.9.7, pytest-6.2.5, py-1.10.0, pluggy-1.0.0 + rootdir: /Users/alenkafrim/repos/arrow/python, configfile: setup.cfg + plugins: hypothesis-6.24.1, lazy-fixture-0.6.3 + collected 204 items + + pyarrow/tests/test_compute.py ................................... [ 46%] + ................................................. [100%] + + ========================= 204 passed in 0.49s ========================== + +.. seealso:: + + For more information about testing see :ref:`testing` section. + +Check styling +------------- + +At the end we also need to check the styling. In Arrow we use a +utility called `Archery <https://arrow.apache.org/docs/developers/archery.html>`_ +to check if code is in line with PEP 8 style guide. + +.. code:: console + + $ archery lint --python --fix + INFO:archery:Running Python formatter (autopep8) + INFO:archery:Running Python linter (flake8) + /Users/alenkafrim/repos/arrow/python/pyarrow/tests/test_compute.py:2288:80: E501 line too long (88 > 79 characters) + +We get a warning that one line is too long. Lets +make the necessary corrections and run it again. + +.. code:: console + + $ archery lint --python --fix + INFO:archery:Running Python formatter (autopep8) + INFO:archery:Running Python linter (flake8) + +Done. Now lets make the Pull Request! + +Creating a Pull Request +----------------------- + +First lets check again the changes made to the local copy +of the Arrow library in the shell. + +.. code:: console + + $ git status + On branch ARROW-14977 + Changes not staged for commit: + (use "git add <file>..." to update what will be committed) + (use "git restore <file>..." to discard changes in working directory) + modified: python/pyarrow/compute.py + modified: python/pyarrow/tests/test_compute.py + + no changes added to commit (use "git add" and/or "git commit -a") + + $ git diff + diff --git a/python/pyarrow/compute.py b/python/pyarrow/compute.py + index 9dac606c3..e8fc775d8 100644 + --- a/python/pyarrow/compute.py + +++ b/python/pyarrow/compute.py + @@ -774,3 +774,45 @@ def bottom_k_unstable(values, k, sort_keys=None, *, memory_pool=None): + sort_keys = map(lambda key_name: (key_name, "ascending"), sort_keys) + options = SelectKOptions(k, sort_keys) + return call_function("select_k_unstable", [values], options, memory_pool) + + + + + +def tutorial_min_max(values, skip_nulls=True): + + """ + + Compute the minimum-1 and maximum-1 values of a numeric array. + + + + Null values are ignored by default. This can be changed through + + ScalarAggregateOptions. + + + + This is a made-up feature for the tutorial purposes. + + + + Parameters + + ---------- + + values : Array + + + + Returns + + ------- + + result : StructScalar of min-1 and max+1 + + + + Examples + + -------- + + >>> import pyarrow.compute as pc + + >>> data = [4, 5, 6, None, 1] + + >>> pc.tutorial_min_max(data) + + <pyarrow.StructScalar: [('min-', 0), ('max+', 7)]> + + """ + + + + options = ScalarAggregateOptions(skip_nulls=skip_nulls) + + min_max = call_function("min_max", [values], options) + + + ... + +Everything looks OK. Now we can make the commit (save our changes +to the branch history): + +.. code:: console + + $ git commit -am "Adding a new compute feature for tutorial purposes" + [ARROW-14977 170ef85be] Adding a new compute feature for tutorial purposes + 2 files changed, 51 insertions(+) + + +check the history of commits + +.. code:: console + + $ git log + commit 170ef85beb8ee629be651e3f93bcc4a69e29cfb8 (HEAD -> ARROW-14977) + Author: Alenka Frim <frim.ale...@gmail.com> + Date: Tue Dec 7 13:45:06 2021 +0100 + + Adding a new compute feature for tutorial purposes + + commit 8cebc4948ab5c5792c20a3f463e2043e01c49828 (master) + Author: Sutou Kouhei <k...@clear-code.com> + Date: Sun Dec 5 15:19:46 2021 +0900 + + ARROW-14981: [CI][Docs] Upload built documents + + We can use this in release process instead of building on release + manager's local environment. + + Closes #11856 from kou/ci-docs-upload + + Authored-by: Sutou Kouhei <k...@clear-code.com> + Signed-off-by: Sutou Kouhei <k...@clear-code.com> + ... + +if necessary we should rebase to upstream master: Review comment: We should probably say when necessary: ```suggestion If you started the branch some time ago, you may need to rebase to upstream master to make sure there are no merge conflicts: ``` ########## File path: docs/source/developers/guide/tutorials/python_tutorial.rst ########## @@ -25,3 +25,513 @@ *************** Python tutorial *************** + +In this tutorial we will make an actual feature contribution to +Arrow following the steps specified by :ref:`quick-ref-guide` +section of the guide and a more detailed :ref:`step_by_step` +section. Navigate there whenever there is some information +you may find is missing here. + +The feature contribution will be added to the compute module +in PyArrow. But you can also follow the steps in case you are +correcting a bug or adding a binding. + +This tutorial is different from the :ref:`step_by_step` as we +will be working on a specific case. This tutorial is not meant +as a step by step guide. + +**Lets start!** + +Set up +------ + +Lets setup the Arrow repository. We presume here that Git is +already installed. Otherwise please see the :ref:`set-up` section. + +Once the `Apache Arrow repository <https://github.com/apache/arrow>`_ +is forked we will clone it and add the link of the main repository +to our upstream. + +.. code:: console + + $ git clone https://github.com/<your username>/arrow.git + $ cd arrow + $ git remote add upstream https://github.com/apache/arrow + +Building PyArrow +---------------- + +Script for building PyArrow differs depending on the Operating +System you are using. For this reason we will only refer to +the instructions for the building process in this tutorial. + +.. seealso:: + + For the **introduction** to the building process refer to the + :ref:`build-arrow-guide` section. + + For the **instructions** on how to build PyArrow refer to the + :ref:`build_pyarrow` section. + +Create a JIRA issue for the new feature +--------------------------------------- + +We will add a new feature that imitates an existing function +``min_max`` from the ``arrow.compute`` module but makes the +interval bigger for 1 in both directions. Note that this is a +made-up function for the purpose of this guide. + +See the example of the ``pc.min_max`` in +`this link <https://arrow.apache.org/cookbook/py/data.html#computing-mean-min-max-values-of-an-array>`_. + +First we need to create a JIRA issue as it doesn't exist yet. +With a JIRA account created we will navigate to the +`Apache Arrow JIRA dashboard <https://issues.apache.org/jira/projects/ARROW>`_ +and click on the **Create** button. + +.. figure:: python_tutorial_jira_title.jpeg + :scale: 70 % + :alt: JIRA dashboard with a window for creating a new issue. + + Creating a JIRA issue, adding title (summary) and components. + +.. figure:: python_tutorial_jira_description.jpeg + :scale: 70 % + :alt: JIRA dashboard with a window for creating a + description for the new issue. + + Creating a JIRA issue, adding a description. + +We will also add some comments to start a conversation. + +.. figure:: python_tutorial_jira_comment.jpeg + :scale: 50 % + :alt: JIRA issue page where comment is being added. + + Adding a comment to the JIRA ticket we created. + +We have successfully created a new JIRA issue with index ARROW-14977. + +.. figure:: python_tutorial_jira_issue.jpeg + :scale: 50 % + :alt: JIRA page of the issue just created. + + Our JIRA issue. Yay! + +To see the issue in JIRA follow +`this link <https://issues.apache.org/jira/browse/ARROW-14977>`_. + +.. seealso:: + + To get more information on JIRA issues go to + :ref:`finding-issues` part of the guide. + +Start the work on a new branch +------------------------------ + +Before we start working on adding the feature we should +create a new branch from updated master. + +.. code:: console + + $ git checkout master + $ git fetch upstream + $ git pull --ff-only upstream master + $ git checkout -b ARROW-14977 + +Lets research the Arrow library to see where the ``pc.min_max`` +function is defined/connected with the C++ and get an idea +where we could implement the new feature. + +.. figure:: python_tutorial_github_search.jpeg + :scale: 50 % + :alt: Apache Arrow GitHub repository dashboard where we are + searching for a pc.min_max function reference. + + We could try to search for the function reference in a + GitHub Apache Arrow repository. + +.. figure:: python_tutorial_github_find_in_file.jpeg + :scale: 50 % + :alt: In the GitHub repository we are searching through the + test_compute.py file for the pc.min_max function. + + And search through the ``test_compute.py`` file in ``pyarrow`` + folder. + +From the search we can see that the function is tested in the +``python/pyarrow/tests/test_compute.py`` file that would mean the +function is defined in the ``compute.py`` file. + +After examining the ``compute.py`` file we can see that together +with ``_compute.py`` the functions from C++ get wrapped into Python. +We will define the new feature at the end of the ``compute.py`` file. + +Lets run some code in the Python console from ``arrow/python`` +directory in order to learn more about ``pc.min_max``. + +.. code:: console + + $ cd python + $ python + + Python 3.9.7 (default, Oct 22 2021, 13:24:00) + [Clang 13.0.0 (clang-1300.0.29.3)] on darwin + Type "help", "copyright", "credits" or "license" for more information. + +We have entered into Python console from the shell and we can +do some research: + +.. code-block:: python + + >>> import pyarrow.compute as pc + >>> data = [4, 5, 6, None, 1] + >>> data + [4, 5, 6, None, 1] + >>> pc.min_max(data) + <pyarrow.StructScalar: [('min', 1), ('max', 6)]> + >>> pc.min_max(data, skip_nulls=False) + <pyarrow.StructScalar: [('min', None), ('max', None)]> + +We will call our new feature ``pc.tutorial_min_max``. We want the +result from our function, that takes the same input data, to be +``[('min-', 0), ('max+', 7)]``. If we specify that the null value should be +included, the result should be equal to ``pc.min_max`` that is +``[('min', None), ('max', None)]``. + +Lets add the first trial code into ``arrow/python/pyarrow/compute.py`` +where we first test the call to the "min_max" function from C++: + +.. code-block:: python + + def tutorial_min_max(values, skip_nulls=True): + """ + Add docstings + + Parameters + ---------- + values : Array + + Returns + ------- + result : TODO + + Examples + -------- + >>> import pyarrow.compute as pc + >>> data = [4, 5, 6, None, 1] + >>> pc.tutorial_min_max(data) + <pyarrow.StructScalar: [('min-', 0), ('max+', 7)]> + """ + + options = ScalarAggregateOptions(skip_nulls=skip_nulls) + return call_function("min_max", [values], options) + +To see if this works we will need to import ``pyarrow.compute`` +again and try: + +.. code-block:: python + + >>> import pyarrow.compute as pc + >>> data = [4, 5, 6, None, 1] + >>> pc.tutorial_min_max(data) + <pyarrow.StructScalar: [('min', 1), ('max', 6)]> + +It’s working. Now we must correct the limits to get the corrected +interval. To do that we have to do some research on +``pyarrow.StructScalar``. In ``test_scalar.py`` under the +``test_struct_duplicate_fields`` we can see an example of how the +``StructScalar`` is created. We could again run the Python console +and try creating one ourselves. + +.. code-block:: python + + >>> import pyarrow as pa + >>> ty = pa.struct([ + ... pa.field('min-', pa.int64()), + ... pa.field('max+', pa.int64()), + ... ]) + >>> pa.scalar([('min-', 3), ('max+', 9)], type=ty) + <pyarrow.StructScalar: [('min-', 3), ('max+', 9)]> + +With the new gained knowledge about ``StructScalar`` and additional +options for the ``pc.min_max`` function we can finish the work. + +.. code-block:: python + + def tutorial_min_max(values, skip_nulls=True): + + """ + Compute the minimum-1 and maximum+1 values of a numeric array. + Null values are ignored by default. This can be changed through + ScalarAggregateOptions. + This is a made-up feature for the tutorial purposes. + Parameters + ---------- + values : Array + Returns + ------- + result : StructScalar of min-1 and max+1 + Examples + -------- + >>> import pyarrow.compute as pc + >>> data = [4, 5, 6, None, 1] + >>> pc.tutorial_min_max(data) + <pyarrow.StructScalar: [('min-', 0), ('max+', 7)]> + """ + + options = ScalarAggregateOptions(skip_nulls=skip_nulls) + min_max = call_function("min_max", [values], options) + + if min_max[0].as_py() is not None: + min_t = min_max[0].as_py()-1 + max_t = min_max[1].as_py()+1 + else: + min_t = min_max[0].as_py() + max_t = min_max[1].as_py() + + ty = pa.struct([ + pa.field('min-', pa.int64()), + pa.field('max+', pa.int64()), + ]) + return pa.scalar([('min-', min_t), ('max+', max_t)], type=ty) + +.. TODO seealso + .. For more information about the Arrow codebase visit + .. :ref:``. (link to working on the Arrow codebase section) + +Adding a test +------------- + +Now we should add a unit test to ``python/pyarrow/tests/test_compute.py`` +and run the pytest. + +.. code-block:: python + + def test_tutorial_min_max(): + arr = [4, 5, 6, None, 1] + l1 = {'min-': 0, 'max+': 7} + l2 = {'min-': None, 'max+': None} + assert pc.tutorial_min_max(arr).as_py() == l1 + assert pc.tutorial_min_max(arr, + skip_nulls=False).as_py() == l2 + +With the unit test added we can run the pytest from the shell. + +.. code:: console + + $ cd python + $ python -m pytest pyarrow/tests/test_compute.py -k test_tutorial_min_max + ======================== test session starts ========================== + platform darwin -- Python 3.9.7, pytest-6.2.5, py-1.10.0, pluggy-1.0.0 + rootdir: /Users/alenkafrim/repos/arrow/python, configfile: setup.cfg + plugins: hypothesis-6.24.1, lazy-fixture-0.6.3 + collected 204 items / 203 deselected / 1 selected + + pyarrow/tests/test_compute.py . [100%] + + ======================== 1 passed, 203 deselected in 0.16s ============ + + + $ python -m pytest pyarrow/tests/test_compute.py + ======================== test session starts =========================== + platform darwin -- Python 3.9.7, pytest-6.2.5, py-1.10.0, pluggy-1.0.0 + rootdir: /Users/alenkafrim/repos/arrow/python, configfile: setup.cfg + plugins: hypothesis-6.24.1, lazy-fixture-0.6.3 + collected 204 items + + pyarrow/tests/test_compute.py ................................... [ 46%] + ................................................. [100%] + + ========================= 204 passed in 0.49s ========================== + +.. seealso:: + + For more information about testing see :ref:`testing` section. + +Check styling +------------- + +At the end we also need to check the styling. In Arrow we use a +utility called `Archery <https://arrow.apache.org/docs/developers/archery.html>`_ +to check if code is in line with PEP 8 style guide. + +.. code:: console + + $ archery lint --python --fix + INFO:archery:Running Python formatter (autopep8) + INFO:archery:Running Python linter (flake8) + /Users/alenkafrim/repos/arrow/python/pyarrow/tests/test_compute.py:2288:80: E501 line too long (88 > 79 characters) + +We get a warning that one line is too long. Lets +make the necessary corrections and run it again. + +.. code:: console + + $ archery lint --python --fix + INFO:archery:Running Python formatter (autopep8) + INFO:archery:Running Python linter (flake8) + +Done. Now lets make the Pull Request! + +Creating a Pull Request +----------------------- + +First lets check again the changes made to the local copy +of the Arrow library in the shell. + +.. code:: console + + $ git status + On branch ARROW-14977 + Changes not staged for commit: + (use "git add <file>..." to update what will be committed) + (use "git restore <file>..." to discard changes in working directory) + modified: python/pyarrow/compute.py + modified: python/pyarrow/tests/test_compute.py + + no changes added to commit (use "git add" and/or "git commit -a") + + $ git diff + diff --git a/python/pyarrow/compute.py b/python/pyarrow/compute.py + index 9dac606c3..e8fc775d8 100644 + --- a/python/pyarrow/compute.py + +++ b/python/pyarrow/compute.py + @@ -774,3 +774,45 @@ def bottom_k_unstable(values, k, sort_keys=None, *, memory_pool=None): + sort_keys = map(lambda key_name: (key_name, "ascending"), sort_keys) + options = SelectKOptions(k, sort_keys) + return call_function("select_k_unstable", [values], options, memory_pool) + + + + + +def tutorial_min_max(values, skip_nulls=True): + + """ + + Compute the minimum-1 and maximum-1 values of a numeric array. + + + + Null values are ignored by default. This can be changed through + + ScalarAggregateOptions. + + + + This is a made-up feature for the tutorial purposes. + + + + Parameters + + ---------- + + values : Array + + + + Returns + + ------- + + result : StructScalar of min-1 and max+1 + + + + Examples + + -------- + + >>> import pyarrow.compute as pc + + >>> data = [4, 5, 6, None, 1] + + >>> pc.tutorial_min_max(data) + + <pyarrow.StructScalar: [('min-', 0), ('max+', 7)]> + + """ + + + + options = ScalarAggregateOptions(skip_nulls=skip_nulls) + + min_max = call_function("min_max", [values], options) + + + ... + +Everything looks OK. Now we can make the commit (save our changes +to the branch history): + +.. code:: console + + $ git commit -am "Adding a new compute feature for tutorial purposes" + [ARROW-14977 170ef85be] Adding a new compute feature for tutorial purposes + 2 files changed, 51 insertions(+) + + +check the history of commits Review comment: ```suggestion You can use `git log` to check the history of commits: ``` ########## File path: docs/source/developers/guide/tutorials/python_tutorial.rst ########## @@ -25,3 +25,513 @@ *************** Python tutorial *************** + +In this tutorial we will make an actual feature contribution to +Arrow following the steps specified by :ref:`quick-ref-guide` +section of the guide and a more detailed :ref:`step_by_step` +section. Navigate there whenever there is some information +you may find is missing here. + +The feature contribution will be added to the compute module +in PyArrow. But you can also follow the steps in case you are +correcting a bug or adding a binding. + +This tutorial is different from the :ref:`step_by_step` as we +will be working on a specific case. This tutorial is not meant +as a step by step guide. + +**Lets start!** + +Set up +------ + +Lets setup the Arrow repository. We presume here that Git is +already installed. Otherwise please see the :ref:`set-up` section. + +Once the `Apache Arrow repository <https://github.com/apache/arrow>`_ +is forked we will clone it and add the link of the main repository +to our upstream. + +.. code:: console + + $ git clone https://github.com/<your username>/arrow.git + $ cd arrow + $ git remote add upstream https://github.com/apache/arrow + +Building PyArrow +---------------- + +Script for building PyArrow differs depending on the Operating +System you are using. For this reason we will only refer to +the instructions for the building process in this tutorial. + +.. seealso:: + + For the **introduction** to the building process refer to the + :ref:`build-arrow-guide` section. + + For the **instructions** on how to build PyArrow refer to the + :ref:`build_pyarrow` section. + +Create a JIRA issue for the new feature +--------------------------------------- + +We will add a new feature that imitates an existing function +``min_max`` from the ``arrow.compute`` module but makes the +interval bigger for 1 in both directions. Note that this is a +made-up function for the purpose of this guide. + +See the example of the ``pc.min_max`` in +`this link <https://arrow.apache.org/cookbook/py/data.html#computing-mean-min-max-values-of-an-array>`_. + +First we need to create a JIRA issue as it doesn't exist yet. +With a JIRA account created we will navigate to the +`Apache Arrow JIRA dashboard <https://issues.apache.org/jira/projects/ARROW>`_ +and click on the **Create** button. + +.. figure:: python_tutorial_jira_title.jpeg + :scale: 70 % + :alt: JIRA dashboard with a window for creating a new issue. + + Creating a JIRA issue, adding title (summary) and components. + +.. figure:: python_tutorial_jira_description.jpeg + :scale: 70 % + :alt: JIRA dashboard with a window for creating a + description for the new issue. + + Creating a JIRA issue, adding a description. + +We will also add some comments to start a conversation. + +.. figure:: python_tutorial_jira_comment.jpeg + :scale: 50 % + :alt: JIRA issue page where comment is being added. + + Adding a comment to the JIRA ticket we created. + +We have successfully created a new JIRA issue with index ARROW-14977. + +.. figure:: python_tutorial_jira_issue.jpeg + :scale: 50 % + :alt: JIRA page of the issue just created. + + Our JIRA issue. Yay! + +To see the issue in JIRA follow +`this link <https://issues.apache.org/jira/browse/ARROW-14977>`_. + +.. seealso:: + + To get more information on JIRA issues go to + :ref:`finding-issues` part of the guide. + +Start the work on a new branch +------------------------------ + +Before we start working on adding the feature we should +create a new branch from updated master. + +.. code:: console + + $ git checkout master + $ git fetch upstream + $ git pull --ff-only upstream master + $ git checkout -b ARROW-14977 + +Lets research the Arrow library to see where the ``pc.min_max`` +function is defined/connected with the C++ and get an idea +where we could implement the new feature. + +.. figure:: python_tutorial_github_search.jpeg + :scale: 50 % + :alt: Apache Arrow GitHub repository dashboard where we are + searching for a pc.min_max function reference. + + We could try to search for the function reference in a + GitHub Apache Arrow repository. + +.. figure:: python_tutorial_github_find_in_file.jpeg + :scale: 50 % + :alt: In the GitHub repository we are searching through the + test_compute.py file for the pc.min_max function. + + And search through the ``test_compute.py`` file in ``pyarrow`` + folder. + +From the search we can see that the function is tested in the +``python/pyarrow/tests/test_compute.py`` file that would mean the +function is defined in the ``compute.py`` file. + +After examining the ``compute.py`` file we can see that together +with ``_compute.py`` the functions from C++ get wrapped into Python. +We will define the new feature at the end of the ``compute.py`` file. + +Lets run some code in the Python console from ``arrow/python`` +directory in order to learn more about ``pc.min_max``. + +.. code:: console + + $ cd python + $ python + + Python 3.9.7 (default, Oct 22 2021, 13:24:00) + [Clang 13.0.0 (clang-1300.0.29.3)] on darwin + Type "help", "copyright", "credits" or "license" for more information. + +We have entered into Python console from the shell and we can +do some research: + +.. code-block:: python + + >>> import pyarrow.compute as pc + >>> data = [4, 5, 6, None, 1] + >>> data + [4, 5, 6, None, 1] + >>> pc.min_max(data) + <pyarrow.StructScalar: [('min', 1), ('max', 6)]> + >>> pc.min_max(data, skip_nulls=False) + <pyarrow.StructScalar: [('min', None), ('max', None)]> + +We will call our new feature ``pc.tutorial_min_max``. We want the +result from our function, that takes the same input data, to be +``[('min-', 0), ('max+', 7)]``. If we specify that the null value should be +included, the result should be equal to ``pc.min_max`` that is +``[('min', None), ('max', None)]``. + +Lets add the first trial code into ``arrow/python/pyarrow/compute.py`` +where we first test the call to the "min_max" function from C++: + +.. code-block:: python + + def tutorial_min_max(values, skip_nulls=True): + """ + Add docstings + + Parameters + ---------- + values : Array + + Returns + ------- + result : TODO + + Examples + -------- + >>> import pyarrow.compute as pc + >>> data = [4, 5, 6, None, 1] + >>> pc.tutorial_min_max(data) + <pyarrow.StructScalar: [('min-', 0), ('max+', 7)]> + """ + + options = ScalarAggregateOptions(skip_nulls=skip_nulls) + return call_function("min_max", [values], options) + +To see if this works we will need to import ``pyarrow.compute`` +again and try: + +.. code-block:: python + + >>> import pyarrow.compute as pc + >>> data = [4, 5, 6, None, 1] + >>> pc.tutorial_min_max(data) + <pyarrow.StructScalar: [('min', 1), ('max', 6)]> + +It’s working. Now we must correct the limits to get the corrected +interval. To do that we have to do some research on +``pyarrow.StructScalar``. In ``test_scalar.py`` under the +``test_struct_duplicate_fields`` we can see an example of how the +``StructScalar`` is created. We could again run the Python console +and try creating one ourselves. + +.. code-block:: python + + >>> import pyarrow as pa + >>> ty = pa.struct([ + ... pa.field('min-', pa.int64()), + ... pa.field('max+', pa.int64()), + ... ]) + >>> pa.scalar([('min-', 3), ('max+', 9)], type=ty) + <pyarrow.StructScalar: [('min-', 3), ('max+', 9)]> + +With the new gained knowledge about ``StructScalar`` and additional +options for the ``pc.min_max`` function we can finish the work. + +.. code-block:: python + + def tutorial_min_max(values, skip_nulls=True): + + """ + Compute the minimum-1 and maximum+1 values of a numeric array. + Null values are ignored by default. This can be changed through + ScalarAggregateOptions. + This is a made-up feature for the tutorial purposes. + Parameters + ---------- + values : Array + Returns + ------- + result : StructScalar of min-1 and max+1 + Examples + -------- + >>> import pyarrow.compute as pc + >>> data = [4, 5, 6, None, 1] + >>> pc.tutorial_min_max(data) + <pyarrow.StructScalar: [('min-', 0), ('max+', 7)]> + """ + + options = ScalarAggregateOptions(skip_nulls=skip_nulls) + min_max = call_function("min_max", [values], options) + + if min_max[0].as_py() is not None: + min_t = min_max[0].as_py()-1 + max_t = min_max[1].as_py()+1 + else: + min_t = min_max[0].as_py() + max_t = min_max[1].as_py() + + ty = pa.struct([ + pa.field('min-', pa.int64()), + pa.field('max+', pa.int64()), + ]) + return pa.scalar([('min-', min_t), ('max+', max_t)], type=ty) + +.. TODO seealso + .. For more information about the Arrow codebase visit + .. :ref:``. (link to working on the Arrow codebase section) + +Adding a test +------------- + +Now we should add a unit test to ``python/pyarrow/tests/test_compute.py`` +and run the pytest. + +.. code-block:: python + + def test_tutorial_min_max(): + arr = [4, 5, 6, None, 1] + l1 = {'min-': 0, 'max+': 7} + l2 = {'min-': None, 'max+': None} + assert pc.tutorial_min_max(arr).as_py() == l1 + assert pc.tutorial_min_max(arr, + skip_nulls=False).as_py() == l2 + +With the unit test added we can run the pytest from the shell. + +.. code:: console + + $ cd python + $ python -m pytest pyarrow/tests/test_compute.py -k test_tutorial_min_max + ======================== test session starts ========================== + platform darwin -- Python 3.9.7, pytest-6.2.5, py-1.10.0, pluggy-1.0.0 + rootdir: /Users/alenkafrim/repos/arrow/python, configfile: setup.cfg + plugins: hypothesis-6.24.1, lazy-fixture-0.6.3 + collected 204 items / 203 deselected / 1 selected + + pyarrow/tests/test_compute.py . [100%] + + ======================== 1 passed, 203 deselected in 0.16s ============ + + + $ python -m pytest pyarrow/tests/test_compute.py + ======================== test session starts =========================== + platform darwin -- Python 3.9.7, pytest-6.2.5, py-1.10.0, pluggy-1.0.0 + rootdir: /Users/alenkafrim/repos/arrow/python, configfile: setup.cfg + plugins: hypothesis-6.24.1, lazy-fixture-0.6.3 + collected 204 items + + pyarrow/tests/test_compute.py ................................... [ 46%] + ................................................. [100%] + + ========================= 204 passed in 0.49s ========================== + +.. seealso:: + + For more information about testing see :ref:`testing` section. + +Check styling +------------- + +At the end we also need to check the styling. In Arrow we use a +utility called `Archery <https://arrow.apache.org/docs/developers/archery.html>`_ +to check if code is in line with PEP 8 style guide. + +.. code:: console + + $ archery lint --python --fix + INFO:archery:Running Python formatter (autopep8) + INFO:archery:Running Python linter (flake8) + /Users/alenkafrim/repos/arrow/python/pyarrow/tests/test_compute.py:2288:80: E501 line too long (88 > 79 characters) + +We get a warning that one line is too long. Lets +make the necessary corrections and run it again. Review comment: Something to note for the reader: the `--fix` command means it will attempt to fix any style issues, but some issues like line length can't be fixed automatically. ########## File path: docs/source/developers/guide/tutorials/python_tutorial.rst ########## @@ -25,3 +25,513 @@ *************** Python tutorial *************** + +In this tutorial we will make an actual feature contribution to +Arrow following the steps specified by :ref:`quick-ref-guide` +section of the guide and a more detailed :ref:`step_by_step` +section. Navigate there whenever there is some information +you may find is missing here. + +The feature contribution will be added to the compute module +in PyArrow. But you can also follow the steps in case you are +correcting a bug or adding a binding. + +This tutorial is different from the :ref:`step_by_step` as we +will be working on a specific case. This tutorial is not meant +as a step by step guide. + +**Lets start!** + +Set up +------ + +Lets setup the Arrow repository. We presume here that Git is +already installed. Otherwise please see the :ref:`set-up` section. + +Once the `Apache Arrow repository <https://github.com/apache/arrow>`_ +is forked we will clone it and add the link of the main repository +to our upstream. + +.. code:: console + + $ git clone https://github.com/<your username>/arrow.git + $ cd arrow + $ git remote add upstream https://github.com/apache/arrow + +Building PyArrow +---------------- + +Script for building PyArrow differs depending on the Operating +System you are using. For this reason we will only refer to +the instructions for the building process in this tutorial. + +.. seealso:: + + For the **introduction** to the building process refer to the + :ref:`build-arrow-guide` section. + + For the **instructions** on how to build PyArrow refer to the + :ref:`build_pyarrow` section. + +Create a JIRA issue for the new feature +--------------------------------------- + +We will add a new feature that imitates an existing function +``min_max`` from the ``arrow.compute`` module but makes the +interval bigger for 1 in both directions. Note that this is a +made-up function for the purpose of this guide. + +See the example of the ``pc.min_max`` in +`this link <https://arrow.apache.org/cookbook/py/data.html#computing-mean-min-max-values-of-an-array>`_. + +First we need to create a JIRA issue as it doesn't exist yet. +With a JIRA account created we will navigate to the +`Apache Arrow JIRA dashboard <https://issues.apache.org/jira/projects/ARROW>`_ +and click on the **Create** button. + +.. figure:: python_tutorial_jira_title.jpeg + :scale: 70 % + :alt: JIRA dashboard with a window for creating a new issue. + + Creating a JIRA issue, adding title (summary) and components. + +.. figure:: python_tutorial_jira_description.jpeg + :scale: 70 % + :alt: JIRA dashboard with a window for creating a + description for the new issue. + + Creating a JIRA issue, adding a description. + +We will also add some comments to start a conversation. + +.. figure:: python_tutorial_jira_comment.jpeg + :scale: 50 % + :alt: JIRA issue page where comment is being added. + + Adding a comment to the JIRA ticket we created. + +We have successfully created a new JIRA issue with index ARROW-14977. + +.. figure:: python_tutorial_jira_issue.jpeg + :scale: 50 % + :alt: JIRA page of the issue just created. + + Our JIRA issue. Yay! + +To see the issue in JIRA follow +`this link <https://issues.apache.org/jira/browse/ARROW-14977>`_. + +.. seealso:: + + To get more information on JIRA issues go to + :ref:`finding-issues` part of the guide. + +Start the work on a new branch +------------------------------ + +Before we start working on adding the feature we should +create a new branch from updated master. + +.. code:: console + + $ git checkout master + $ git fetch upstream + $ git pull --ff-only upstream master + $ git checkout -b ARROW-14977 + +Lets research the Arrow library to see where the ``pc.min_max`` +function is defined/connected with the C++ and get an idea +where we could implement the new feature. + +.. figure:: python_tutorial_github_search.jpeg + :scale: 50 % + :alt: Apache Arrow GitHub repository dashboard where we are + searching for a pc.min_max function reference. + + We could try to search for the function reference in a + GitHub Apache Arrow repository. + +.. figure:: python_tutorial_github_find_in_file.jpeg + :scale: 50 % + :alt: In the GitHub repository we are searching through the + test_compute.py file for the pc.min_max function. + + And search through the ``test_compute.py`` file in ``pyarrow`` + folder. + +From the search we can see that the function is tested in the +``python/pyarrow/tests/test_compute.py`` file that would mean the +function is defined in the ``compute.py`` file. + +After examining the ``compute.py`` file we can see that together +with ``_compute.py`` the functions from C++ get wrapped into Python. +We will define the new feature at the end of the ``compute.py`` file. + +Lets run some code in the Python console from ``arrow/python`` +directory in order to learn more about ``pc.min_max``. + +.. code:: console + + $ cd python + $ python + + Python 3.9.7 (default, Oct 22 2021, 13:24:00) + [Clang 13.0.0 (clang-1300.0.29.3)] on darwin + Type "help", "copyright", "credits" or "license" for more information. + +We have entered into Python console from the shell and we can +do some research: + +.. code-block:: python + + >>> import pyarrow.compute as pc + >>> data = [4, 5, 6, None, 1] + >>> data + [4, 5, 6, None, 1] + >>> pc.min_max(data) + <pyarrow.StructScalar: [('min', 1), ('max', 6)]> + >>> pc.min_max(data, skip_nulls=False) + <pyarrow.StructScalar: [('min', None), ('max', None)]> + +We will call our new feature ``pc.tutorial_min_max``. We want the +result from our function, that takes the same input data, to be +``[('min-', 0), ('max+', 7)]``. If we specify that the null value should be +included, the result should be equal to ``pc.min_max`` that is +``[('min', None), ('max', None)]``. + +Lets add the first trial code into ``arrow/python/pyarrow/compute.py`` +where we first test the call to the "min_max" function from C++: + +.. code-block:: python + + def tutorial_min_max(values, skip_nulls=True): + """ + Add docstings + + Parameters + ---------- + values : Array + + Returns + ------- + result : TODO + + Examples + -------- + >>> import pyarrow.compute as pc + >>> data = [4, 5, 6, None, 1] + >>> pc.tutorial_min_max(data) + <pyarrow.StructScalar: [('min-', 0), ('max+', 7)]> + """ + + options = ScalarAggregateOptions(skip_nulls=skip_nulls) + return call_function("min_max", [values], options) + +To see if this works we will need to import ``pyarrow.compute`` +again and try: + +.. code-block:: python + + >>> import pyarrow.compute as pc + >>> data = [4, 5, 6, None, 1] + >>> pc.tutorial_min_max(data) + <pyarrow.StructScalar: [('min', 1), ('max', 6)]> + +It’s working. Now we must correct the limits to get the corrected +interval. To do that we have to do some research on +``pyarrow.StructScalar``. In ``test_scalar.py`` under the +``test_struct_duplicate_fields`` we can see an example of how the +``StructScalar`` is created. We could again run the Python console Review comment: Might be worth adding a permalink to the code: ```https://github.com/apache/arrow/blob/994074d2e7ff073301e0959dbc5bb595a1e2a41b/python/pyarrow/tests/test_scalars.py#L548-L552``` Also: ```suggestion ``pyarrow.StructScalar``. In ``test_scalars.py`` under the ``test_struct_duplicate_fields`` we can see an example of how the ``StructScalar`` is created. We could again run the Python console ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@arrow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
