[
https://issues.apache.org/jira/browse/ARROW-12099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17541192#comment-17541192
]
Guido Muscioni edited comment on ARROW-12099 at 5/23/22 10:21 PM:
------------------------------------------------------------------
(sorry for continuing this old thread, I can open a new issue if necessary)
I was working on [~jorisvandenbossche] implementation and noticed that it skips
the null values. I slightly modified the code to handle missing values in a
list:
{code:python}
def explode_table_include_null(table, column):
other_columns = list(table.schema.names)
other_columns.remove(column)
indices = pc.list_parent_indices(pc.fill_null(table[column], [None]))
result = table.select(other_columns).take(indices)
result = result.append_column(
pa.field(column, table.schema.field(column).type.value_type),
pc.list_flatten(pc.fill_null(table[column], [None])),
)
return result
{code}
For example, given this table:
{code:python}
table = pa.table({'a': range(3), 'b': [[1,2], None, [1,2,3]]})
{code}
pandas in this case will leave the line with the None value:
{code:python}
In [14]: table.to_pandas().explode('b')
Out[14]:
a b
0 0 1
0 0 2
1 1 None
2 2 1
2 2 2
2 2 3
{code}
[~jorisvandenbossche] implementation will not return the None row:
{code:python}
In [16]: explode_table(table, 'b').to_pandas()
Out[16]:
a b
0 0 1
1 0 2
2 2 1
3 2 2
4 2 3
{code}
Using the function above:
{code:python}
In [18]: explode_table_include_null(table, 'b').to_pandas()
Out[18]:
a b
0 0 1.0
1 0 2.0
2 1 NaN
3 2 1.0
4 2 2.0
5 2 3.0
{code}
I am not sure if filling the value with a blind empty list is correct, it seems
to work well given multiple examples (I am referring to the requirements of the
pc.fill_null function about typing):
{code:python}
In [19]: table = pa.table({'a': range(3), 'b': [['1','2'],None, ['1','2','3']]})
In [20]: explode_table_include_null(table, 'b').to_pandas()
Out[20]:
a b
0 0 1
1 0 2
2 1 None
3 2 1
4 2 2
5 2 3
In [22]: table = pa.table({'a': range(3), 'b': [[{'a':1}],None, [{'a':1}]]})
In [23]: explode_table_include_null(table, 'b').to_pandas()
Out[23]:
a b
0 0 {'a': 1.0}
1 1 None
2 2 {'a': 1.0}
{code}
I did a very small benchmarking of this function against pandas and the speedup
is remarkable:
{code:java}
In [25]: %timeit explode_table_include_null(table, 'b')
131 µs ± 1.9 µs per loop (mean ± std. dev. of 7 runs, 10,000 loops each)
In [26]: %timeit df.explode('b')
3.78 ms ± 69.1 µs per loop (mean ± std. dev. of 7 runs, 100 loops each){code}
was (Author: muscionig):
(sorry for continuing this old thread, I can open a new issue if necessary)
I was working on [~jorisvandenbossche] implementation and noticed that it skips
the null values. I slightly modified the code to handle missing values in a
list:
{code:python}
def explode_table_include_null(table, column):
other_columns = list(table.schema.names)
other_columns.remove(column)
indices = pc.list_parent_indices(pc.fill_null(table[column], [None]))
result = table.select(other_columns).take(indices)
result = result.append_column(
pa.field(column, table.schema.field(column).type.value_type),
pc.list_flatten(pc.fill_null(table[column], [None])),
)
return result
{code}
For example, given this table:
{code:python}
table = pa.table({'a': range(3), 'b': [[1,2], None, [1,2,3]]})
{code}
pandas in this case will leave the line with the None value:
{code:python}
In [14]: table.to_pandas().explode('b')
Out[14]:
a b
0 0 1
0 0 2
1 1 None
2 2 1
2 2 2
2 2 3
{code}
[~jorisvandenbossche] implementation will not return the None row:
{code:python}
In [16]: explode_table(table, 'b').to_pandas()
Out[16]:
a b
0 0 1
1 0 2
2 2 1
3 2 2
4 2 3
{code}
Using the function above:
{code:python}
In [18]: explode_table_include_null(table, 'b').to_pandas()
Out[18]:
a b
0 0 1.0
1 0 2.0
2 1 NaN
3 2 1.0
4 2 2.0
5 2 3.0
{code}
I am not sure if filling the value with a blind empty list is correct, it seems
to work well given multiple examples (I am referring to the requirements of the
pc.fill_null function about typing):
{code:python}
In [19]: table = pa.table({'a': range(3), 'b': [['1','2'],None, ['1','2','3']]})
In [20]: explode_table_include_null(table, 'b').to_pandas()
Out[20]:
a b
0 0 1
1 0 2
2 1 None
3 2 1
4 2 2
5 2 3
In [22]: table = pa.table({'a': range(3), 'b': [[{'a':1}],None, [{'a':1}]]})
In [23]: explode_table_include_null(table, 'b').to_pandas()
Out[23]:
a b
0 0 {'a': 1.0}
1 1 None
2 2 {'a': 1.0}
{code}
I did a very small benchmarking of this function against pandas and the speedup
is remarkable:
```
In [25]: %timeit explode_table_include_null(table, 'b')
131 µs ± 1.9 µs per loop (mean ± std. dev. of 7 runs, 10,000 loops each)
In [26]: %timeit df.explode('b')
3.78 ms ± 69.1 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
```
> [Python] Explode array column
> -----------------------------
>
> Key: ARROW-12099
> URL: https://issues.apache.org/jira/browse/ARROW-12099
> Project: Apache Arrow
> Issue Type: New Feature
> Components: Python
> Reporter: Malthe Borch
> Priority: Major
>
> In Apache Spark,
> [explode|https://spark.apache.org/docs/latest/api/sql/index.html#explode]
> separates the elements of an array column (or expression) into multiple row.
> Note that each explode works at the top-level only (not recursively).
> This would also work with the existing
> [flatten|https://arrow.apache.org/docs/python/generated/pyarrow.Table.html#pyarrow.Table.flatten]
> method to allow fully unnesting a
> [pyarrow.StructArray|https://arrow.apache.org/docs/python/generated/pyarrow.StructArray.html#pyarrow-structarray].
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
