[
https://issues.apache.org/jira/browse/BEAM-7990?focusedWorklogId=296441&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-296441
]
ASF GitHub Bot logged work on BEAM-7990:
----------------------------------------
Author: ASF GitHub Bot
Created on: 16/Aug/19 17:30
Start Date: 16/Aug/19 17:30
Worklog Time Spent: 10m
Work Description: TheNeuralBit commented on pull request #9361:
[BEAM-7990] Add ability to read parquet files into PCollection<pyarrow.Table>
URL: https://github.com/apache/beam/pull/9361
Adds `ReadFromParquetBatched` and `ReadAllFromParquetBatched` to
`apache_beam.io.parquetio`.
This moves the logic to split pyarrow.Table instances into python
dictionaries out of `_ParquetSource` and into a new
`_ArrowTableToRowDictionaries(DoFn)`. The original `ReadFromParquet` expands to
`Read(source) | ParDo(_ArrowTableToRowDictionaries)` while
`ReadFromParquetBatched` just reads directly from the source.
Because many of the tests operate on the source directly and not the actual
IO transforms, they had to be modified to compare the output with the expected
pyarrow Tables. This also created an issue with
`apache_beam.testing.util.equal_to` - because `pyarrow.Table` instances don't
sort deterministically, the faster comparison approach doesn't work correctly.
This PR also modifies `equal_to` to verify inequality with the "slower"
approach. My reasoning is that most test results will be a true positive, and
just the faster approach can be used, so this shouldn't have a substantial
effect on performance.
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/)
| --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/)
| --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/)
| --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/)
| --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/)
XLang | --- | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/)
| --- | --- | ---
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://builds.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/)
| --- | ---
See
[.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md)
for trigger phrase, status and link of all Jenkins jobs.
R: @ihji
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 296441)
Time Spent: 10m
Remaining Estimate: 0h
> Add ability to read parquet files into PCollection<pyarrow.Table>
> -----------------------------------------------------------------
>
> Key: BEAM-7990
> URL: https://issues.apache.org/jira/browse/BEAM-7990
> Project: Beam
> Issue Type: New Feature
> Components: io-py-parquet
> Reporter: Brian Hulette
> Assignee: Brian Hulette
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
