[
https://issues.apache.org/jira/browse/BEAM-214?focusedWorklogId=81900&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-81900
]
ASF GitHub Bot logged work on BEAM-214:
---------------------------------------
Author: ASF GitHub Bot
Created on: 19/Mar/18 15:58
Start Date: 19/Mar/18 15:58
Worklog Time Spent: 10m
Work Description: Igosuki commented on issue #1851: [BEAM-214] Add
ParquetIO
URL: https://github.com/apache/beam/pull/1851#issuecomment-374264085
I already tested this code in production, and another approach using Hadoop
input file IO and it works but not as well as delimiter based file IOs.
We should be able to create splits based on row groups.
I think what you need is something leveraging the latest features from S3A
(fadvise, head skip) and splits with hints for row groups, and obviously
including predicates and projections.
Right now for me, it's not satisfactory because with large parquet files
batch processes quickly become bottlenecked and parallelism is very
limited. It's frustrating that Spark is still better at this.
On Mon, Mar 19, 2018, 4:23 PM Jean-Baptiste Onofré <[email protected]>
wrote:
> For all watchers on this PR, we have basically three approaches:
>
> 1. If you take a look on the first commits in the PR, ParquetIO didn't
> leverage FileIO (because it didn't exist in the early stage). It was a
> complete IO with the "split" was limited (even if it worked fine for
both
> read and write).
> 2. To be able to leverage FileIO, we need a way to use NIO Channel for
> Read. That's possible (via InputStream) with Parquet master as they
> expose the API (take a look on the mail I sent on the dev mailing list
for
> details). This approach requires a new Parquet release.
> 3. Write our own Parquet reader/writer (low level, a bit like we did
> for Avro). It requires more work and I'm not so comfortable with this
> approach especially to leverage features and fixes provided by Parquet.
>
> So, basically, I would love to hear your thoughts about ParquetIO: would
> you like a first simple ParquetIO in Beam even if the split is not
> optimal or if it's not based on FileIO, or are you OK for wait to have a
> FileBased one ?
>
> By the way, I'm resuming my work on this one now that I'm back from
> travels.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/beam/pull/1851#issuecomment-374251378>, or mute
> the thread
>
<https://github.com/notifications/unsubscribe-auth/AAYyhiUODG1Ql0EdRAJa9HkBxIK3nNyPks5tf82AgaJpZM4LupIS>
> .
>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 81900)
Time Spent: 3h 10m (was: 3h)
> Create Parquet IO
> -----------------
>
> Key: BEAM-214
> URL: https://issues.apache.org/jira/browse/BEAM-214
> Project: Beam
> Issue Type: Improvement
> Components: io-ideas
> Reporter: Neville Li
> Assignee: Jean-Baptiste Onofré
> Priority: Minor
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> Would be nice to support Parquet files with projection and predicates.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
