hinxx commented on issue #34262:
URL: https://github.com/apache/arrow/issues/34262#issuecomment-1439573572
Here is the script:
```
import pandas as pd
import pyarrow as pa
from pyarrow import orc
import json
names = pd.Series(['row1', 'row2', 'row3', 'row4', 'row5', 'row6'],
dtype='string', name='name')
timestamps = pd.Series([pd.Timestamp('2022-07-15 12:40:13.439549952'),
pd.Timestamp('2022-07-15 12:40:13.439546880'), pd.Timestamp('2023-02-08
09:13:32.287076352'), pd.Timestamp('2023-02-08 09:13:32.587076352'),
pd.Timestamp('2022-07-07 14:23:10.092787968'), pd.Timestamp('2022-07-15
12:40:13.839546624')], dtype='datetime64[ns]', name='timestamp')
tags = pd.Series([1, 1, 0, 0, 2, 1], dtype='uint8', name='tag')
offsets = pd.Series([0, 1, 0, 1, 0, 2], dtype='uint32', name='offset')
integers = pd.Series([5, 53], dtype='int64', name='integer')
floats = pd.Series([0.011021, -32580.0, -33580.0], dtype='float64',
name='float')
strings = pd.Series(['3.10.0'], dtype='string', name='string')
union_schema = pa.union([
pa.field('int64', pa.int64()),
pa.field('float64', pa.float64()),
pa.field('string', pa.string())
], 'dense')
schema = pa.schema([
('name', pa.string()),
('timestamp', pa.timestamp('ns')),
('value', union_schema)
])
union = pa.UnionArray.from_dense(
pa.array(tags, type='int8'),
pa.array(offsets, type='int32'),
[ pa.Array.from_pandas(integers),
pa.Array.from_pandas(floats),
pa.Array.from_pandas(strings)
],
['int64', 'float64', 'string']
)
table = pa.Table.from_arrays([
pa.Array.from_pandas(names),
pa.Array.from_pandas(timestamps),
union
], schema=schema)
print('table', table)
writer = orc.ORCWriter('union1.orc', dictionary_key_size_threshold=1)
writer.write(table)
```
And this the output I get:
```
$ python union1.py
table pyarrow.Table
name: string
timestamp: timestamp[ns]
value: dense_union<int64: int64=0, float64: double=1, string: string=2>
child 0, int64: int64
child 1, float64: double
child 2, string: string
----
name: [["row1","row2","row3","row4","row5","row6"]]
timestamp: [[2022-07-15 12:40:13.439549952,2022-07-15
12:40:13.439546880,2023-02-08 09:13:32.287076352,2023-02-08
09:13:32.587076352,2022-07-07 14:23:10.092787968,2022-07-15 12:40:13.839546624]]
value: [ -- is_valid: all not null -- type_ids: [1,1,0,0,2,1] --
value_offsets: [0,1,0,1,0,2]
-- child 0 type: int64
[5,53]
-- child 1 type: double
[0.011021,-32580,-33580]
-- child 2 type: string
["3.10.0"]]
Traceback (most recent call last):
File "union1.py", line 42, in <module>
writer.write(table)
File
"/data/data/Code/orc/python/venv/lib/python3.8/site-packages/pyarrow/orc.py",
line 289, in write
self.writer.write(table)
File "pyarrow/_orc.pyx", line 443, in pyarrow._orc.ORCWriter.write
File "pyarrow/error.pxi", line 121, in pyarrow.lib.check_status
pyarrow.lib.ArrowNotImplementedError: Unknown or unsupported Arrow type:
dense_union<int64: int64=0, float64: double=1, string: string=2>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
