[
https://issues.apache.org/jira/browse/ARROW-16641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17544604#comment-17544604
]
Will Jones commented on ARROW-16641:
------------------------------------
{quote}It would be pretty cool if one day `arrow` would have a built-in
function to perform such a task on the fly!
{quote}
Yes, I agree. I believe what you are looking for is the {{arrays_overlap()}}
function in Spark or the {{&&}} operator in PostgreSQL, so I've created a new
ticket to implement that: https://issues.apache.org/jira/browse/ARROW-16702
{quote}So to prepare an "exploded table", we need to load the full dataset into
memory.
{quote}
I'd also add that for cases like this, {{map_batches()}} will one day be a good
solution. Right now it collects its results fully into memory, but we should
eventually let it stream out the results. For now, it does process the _inputs_
one-by-one, so it works well for reducing/aggregating.
> [R] How to filter array columns?
> --------------------------------
>
> Key: ARROW-16641
> URL: https://issues.apache.org/jira/browse/ARROW-16641
> Project: Apache Arrow
> Issue Type: Wish
> Components: R
> Reporter: Vladimir
> Priority: Minor
> Fix For: 9.0.0
>
>
> In the parquet data we have, there is a column with the array data type
> ({*}list<array_element <string>>{*}), which flags records that have different
> issues. For each record, multiple values could be stored in the column. For
> example, `{_}[A, B, C]{_}`.
> I'm trying to perform a data filtering step and exclude some flagged records.
> Filtering is trivial for the regular columns that contain just a single
> value. E.g.,
> {code:java}
> flags_to_exclude <- c("A", "B")
> datt %>% filter(! col %in% flags_to_exclude)
> {code}
> Given the array column, is it possible to exclude records with at least one
> of the flags from `flags_to_exclude` using the arrow R package?
> I really appreciate any advice you can provide!
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
