[GitHub] spark pull request #23021: [SPARK-26032][PYTHON] Break large sql/tests.py fi...

[HyukjinKwon]

GitHub user HyukjinKwon opened a pull request:

    https://github.com/apache/spark/pull/23021

    [SPARK-26032][PYTHON] Break large sql/tests.py files into smaller files

    ## What changes were proposed in this pull request?
    
    This is the official first attempt to break huge single `tests.py` file - I 
did it locally before few times and gave up for some reasons. Now, currently it 
really makes the unittests super hard to read and difficult to check. To me, it 
even bothers me to to scroll down the big file. It's one single 7000 lines file!
    
    This is not only readability issue. Since one big test takes most of tests 
time, the tests don't run in parallel fully, causing the test time increases.
    
    We could pick up one example and follow. Given my investigation, the 
current style looks closer to NumPy structure and looks easier to follow. 
Please see https://github.com/numpy/numpy/tree/master/numpy.
    
    Basically this PR proposes to break down `pyspark/sql/tests.py` into ...:
    
    ```bash
    pyspark
    ...
    ├── sql
    ...
    │   ├── tests  # Includes all tests broken down from 
'pyspark/sql/tests.py'
    │   │   │      # Each matchs to module in 'pyspark/sql'. 
Additionally, some logical group can be added.
    │   │   │      # For instance, 'test_arrow.py', 
'test_datasources.py', 'test_serde.py' and 'test_submit.py'
    │   │   │      # were added.
    │   │   ├── __init__.py
    │   │   ├── test_arrow.py
    │   │   ├── test_catalog.py
    │   │   ├── test_column.py
    │   │   ├── test_conf.py
    │   │   ├── test_context.py
    │   │   ├── test_dataframe.py
    │   │   ├── test_datasources.py
    │   │   ├── test_functions.py
    │   │   ├── test_group.py
    │   │   ├── test_readwriter.py
    │   │   ├── test_serde.py
    │   │   ├── test_session.py
    │   │   ├── test_streaming.py
    │   │   ├── test_submit.py
    │   │   ├── test_types.py
    │   │   ├── test_udf.py
    │   │   └── test_utils.py
    ...
    ├── testing  # Includes testing utils that can be used in unittests.
    │   ├── __init__.py
    │   └── sqlutils.py
    ...
    ```
    
    ## How was this patch tested?
    
    Existing tests should cover.
    
    `cd python` and `./run-tests-with-coverage`. Manually checked they are 
actually being ran.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/HyukjinKwon/spark SPARK-25344

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/23021.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #23021
    
----
commit dee09b73757bbad73cbe274d6a9873d8cc143423
Author: hyukjinkwon <gurwls223@...>
Date:   2018-11-13T08:41:57Z

    Break larga sql/tests.py files into smaller files

----


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org