Re: DSV2 API Question

2019-06-27 Thread Bobby Evans
Columnar UDFs is still a work in progress. For now, all UDFs are row-based, and in fact, all processing is row based. We are working on plumbing in more columnar support to Spark https://github.com/apache/spark/pull/24795 but it is going to be a little while before we are at the point where we

DSV2 API Question

2019-06-25 Thread Andrew Melo
Hello, I've (nearly) implemented a DSV2-reader interface to read particle physics data stored in the ROOT (https://root.cern.ch/) file format. You can think of these ROOT files as roughly parquet-like: column-wise and nested (i.e. a column can be of type "float[]", meaning each row in the column