Hi Hongze, To add to Wes's point, there are already some efforts to do JNI for ORC (which needs to be integrated with CI) and some open PRs for Parquet in the project. However, given that you are using Spark I would expect there is already dataset functionality that is equivalent to the dataset API to do rowgroup/partition level filtering. Can you elaborate on what problems you are seeing with those and what additional use cases you have?
Thanks, Micah On Tue, Nov 26, 2019 at 1:10 PM Wes McKinney <wesmck...@gmail.com> wrote: > hi Hongze, > > The Datasets functionality is indeed extremely useful, and it may make > sense to have it available in many languages eventually. With Java, I > would raise the issue that things are comparatively weaker there when > it comes to actually reading the files themselves. Whereas we have > reasonably fast Arrow-based interfaces to CSV, JSON, ORC, and Parquet > in C++ the same is not true in Java. Not a deal breaker but worth > taking into consideration. > > I wonder aloud whether it might be worth investing in a JNI-based > interface to the C++ libraries as one potential approach to save on > development time. > > - Wes > > > > On Tue, Nov 26, 2019 at 5:54 AM Hongze Zhang <notify...@126.com> wrote: > > > > Hi all, > > > > > > Recently the datasets API has been improved a lot and I found some of > the new features are very useful to my own work. For example to me a > important one is the fix of ARROW-6952[1]. And as I currently work on > Java/Scala projects like Spark, I am now investigating a way to call some > of the datasets APIs in Java so that I could gain performance improvement > from native dataset filters/projectors. Meantime I am also interested in > the ability of scanning different data sources provided by dataset API. > > > > > > Regarding using datasets in Java, my initial idea is to port (by writing > Java-version implementations) some of the high-level concepts in Java such > as DataSourceDiscovery/DataSet/Scanner/FileFormat, then create and call > lower level record batch iterators via JNI. This way we seem to retain > performance advantages from c++ dataset code. > > > > > > Is anyone interested in this topic also? Or is this something already on > the development plan? Any feedback or thoughts would be much appreciated. > > > > > > Best, > > Hongze > > > > > > [1] https://issues.apache.org/jira/browse/ARROW-6952 >
