[
https://issues.apache.org/jira/browse/PARQUET-218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14366667#comment-14366667
]
Alex Levenson commented on PARQUET-218:
---------------------------------------
One more thing we need to support is projecting / filtering into repeated
columns, which is a little bit harder.
A normal function for that would look like:
myStruct.foo.bar.someList.map { x => (x.a, x.d, x.f) }
And for filtering:
myStruct.foo.bar.someList.filter { x => x.a > 0 && x.d > 0 }
So these macros will have to recurse, instead of taking the approach of pulling
the 0th element out of a repeated field and operating on that.
> Macro based Scala APIs for Predicate and Projection push down
> -------------------------------------------------------------
>
> Key: PARQUET-218
> URL: https://issues.apache.org/jira/browse/PARQUET-218
> Project: Parquet
> Issue Type: Improvement
> Components: parquet-mr
> Reporter: Julien Le Dem
>
> There has been some experiments with Scala macros to make Predicate and
> Projection APIs in Scala.
> In particular by [~neville]:
> https://github.com/nevillelyh/parquet-avro-extra/blob/master/parquet-avro-examples/src/main/scala/me/lyh/parquet/avro/examples/ExampleApp.scala
> https://github.com/nevillelyh/parquet-avro-extra/commit/090f8d1d1d7d44489bb71e04465d4a12a46432c9#diff-89dca6adcdf105c6063661de43931c29R15
> There have also been some discussion at Twitter.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
