[
https://issues.apache.org/jira/browse/ARROW-7945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17140365#comment-17140365
]
Niklas B commented on ARROW-7945:
---------------------------------
[~jorisvandenbossche] That's a good point! I have tried using numpy to create a
boolean mask and while that operations is zero copy and lightning fast, the
actual filter takes a fair bit of time (2 seconds) to filter out about 50% of
700k rows (~200 columns). I'm guessing its iterating each column, where each
column takes about 0.01s. Possibly I can improve this if I write a function in
cython/c++, but it's slightly out of my abilities. I'll probably wait for the
Dataset API :)
> [C++][Dataset] Implement InMemoryDatasetFactory
> -----------------------------------------------
>
> Key: ARROW-7945
> URL: https://issues.apache.org/jira/browse/ARROW-7945
> Project: Apache Arrow
> Issue Type: Improvement
> Components: C++
> Affects Versions: 0.16.0
> Reporter: Ben Kietzman
> Assignee: Ben Kietzman
> Priority: Major
> Labels: dataset
>
> This will allow in memory datasets (such as tables) to participate in
> discovery through {{UnionDatasetFactory}}. This class will be trivial since
> Inspect will do nothing but return the table's schema, but is necessary to
> ensure that the resulting {{UnionDataset}}'s unified schema accommodates the
> table's schema (for example including fields present only in the table's
> schema or emitting an error when unification is not possible)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
