[
https://issues.apache.org/jira/browse/BEAM-214?focusedWorklogId=81880&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-81880
]
ASF GitHub Bot logged work on BEAM-214:
---------------------------------------
Author: ASF GitHub Bot
Created on: 19/Mar/18 15:23
Start Date: 19/Mar/18 15:23
Worklog Time Spent: 10m
Work Description: jbonofre commented on issue #1851: [BEAM-214] Add
ParquetIO
URL: https://github.com/apache/beam/pull/1851#issuecomment-374251378
For all watchers on this PR, we have basically three approaches:
1. If you take a look on the first commits in the PR, `ParquetIO` didn't
leverage `FileIO` (because it didn't exist in the early stage). It was a
complete IO with the "split" was limited (even if it worked fine for both read
and write).
2. To be able to leverage `FileIO`, we need a way to use NIO Channel for
Read. That's possible (via `InputStream`) with Parquet master as they expose
the API (take a look on the mail I sent on the dev mailing list for details).
This approach requires a new Parquet release.
3. Write our own Parquet reader/writer (low level, a bit like we did for
Avro). It requires more work and I'm not so comfortable with this approach
especially to leverage features and fixes provided by Parquet.
So, basically, I would love to hear your thoughts about `ParquetIO`: would
you like a first simple `ParquetIO` in Beam even if the split is not optimal or
if it's not based on `FileIO`, or are you OK for wait to have a `FileBased` one
?
By the way, I'm resuming my work on this one now that I'm back from travels.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 81880)
Time Spent: 3h (was: 2h 50m)
> Create Parquet IO
> -----------------
>
> Key: BEAM-214
> URL: https://issues.apache.org/jira/browse/BEAM-214
> Project: Beam
> Issue Type: Improvement
> Components: io-ideas
> Reporter: Neville Li
> Assignee: Jean-Baptiste Onofré
> Priority: Minor
> Time Spent: 3h
> Remaining Estimate: 0h
>
> Would be nice to support Parquet files with projection and predicates.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
