I have been thinking about this project more, and the more I think about it the more I like it.
For example of the kind of leverage a library like this might bring, we might consider changing the implementation of Arrow UDF to re-use the underlying buffers when possible (e.g. via unary_mut[1]). This would likely provide an across the board efficiency improvement for no costs to downstream crates. Andrew [1]: https://docs.rs/arrow/latest/arrow/array/struct.PrimitiveArray.html#method.unary_mut On Sat, Jun 29, 2024 at 1:47 AM Xuanwo <xua...@apache.org> wrote: > > That said, wherever it ends up, there should be the agreement of > > individuals to accept maintenance of it. Since it's in rust, that would > > generally fall to the arrow-rs contributors and/or the DataFusion > > contributors IMO. > > > > It would be good for it to be part of the community, but only if it's not > > going to end up just bitrotting somewhere. > > Thanks Matt. This concern does make sense. > > Arrow UDF is extensively used within RisingWave and Databend. We, the > initial > committers from both RisingWave and Databend, are eager to take > responsibility > for maintaining these crates. > > Additionally, some of us are involved in other Apache Projects, so we > understand > how the Apache Way functions. We will focus on community growth to ensure > this > project remains active. > > On Sat, Jun 29, 2024, at 13:29, Matt Topol wrote: > >> This UDF implementation doesn’t depend on DataFusion. It can work with > > any data in the arrow format. > > > > Given this I'm in agreement with Antoine that it would be weird for it to > > be maintained within the DataFusion repo as opposed to it's own repo (as > > we've done in the past for things like nanoarrow and arrow-experiments). > > > > That said, wherever it ends up, there should be the agreement of > > individuals to accept maintenance of it. Since it's in rust, that would > > generally fall to the arrow-rs contributors and/or the DataFusion > > contributors IMO. > > > > It would be good for it to be part of the community, but only if it's not > > going to end up just bitrotting somewhere. > > > > --Matt > > > > On Fri, Jun 28, 2024, 8:49 PM Xuanwo <xua...@apache.org> wrote: > > > >> Hi, > >> > >> This UDF implementation doesn’t depend on DataFusion. It can work with > any > >> data in the arrow format. > >> > >> It has the potential power to make users write ONE UDF function that > works > >> for different query engines as we showed up in databend and risingwave. > >> > >> So I personally think it should be part of arrow community. > >> > >> On Sat, Jun 29, 2024, at 05:06, Antoine Pitrou wrote: > >> > Is this UDF implementation based on DataFusion? If so, it makes sense > >> > for it to be part of the DataFusion project. > >> > > >> > OTOH, if it can work with any data in the Arrow format, then it would > >> > sound weird to maintain it in the DataFusion repo IMHO. > >> > > >> > Regards > >> > > >> > Antoine. > >> > > >> > > >> > Le 28/06/2024 à 21:52, Andrew Lamb a écrit : > >> >> To be clear, if the arrow community thinks this would be better > >> organized / > >> >> administered in the Apache DataFusion project (especially if it is > >> aligned > >> >> with Rust) I think it would be good to discuss donating there > >> >> > >> >> On Fri, Jun 28, 2024 at 3:17 PM Andrew Lamb <al...@influxdata.com> > >> wrote: > >> >> > >> >>> I think there are two aspects: > >> >>> 1. The actual mechanics of implementing functions > >> >>> 2. The actual library of udf functions (e.g. sin, cos, nullif, etc) > >> >>> > >> >>> I agree 2 is not something that belongs naturally in the arrow > project > >> and > >> >>> is better aligned with query engines > >> >>> > >> >>> However I think 1 is worth considering. > >> >>> > >> >>> As I understand it, the problem arrow_udf solves is avoiding some of > >> the > >> >>> boilerplate required to make vectorized udfs. So instead of > writing a > >> >>> special eval_gcd function like this > >> >>> > >> >>> ``` > >> >>> fn gcd(l: i64, r: i64) -> i64 { > >> >>> // do gcd calculation > >> >>> } > >> >>> > >> >>> // implement vectorized version > >> >>> fn eval_gcd(left: &ArrayRef, right: &ArrayRef) -> ArrayRef { > >> >>> let left = left.as_primitive<Int64Type>(); > >> >>> let right = right.as_primitive<Int64Type>(); > >> >>> res = binary(left, right, |l, r| gcd(l, r)); > >> >>> Arc::new(res) > >> >>> } > >> >>> ``` > >> >>> > >> >>> The user simply annotates the scalar function and have the library > code > >> >>> gen the array version > >> >>> ``` > >> >>> #[function("gcd(int64, int64) -> int64", output = "eval_gcd")] > >> >>> fn gcd(l: i64, r: i64) -> i64 { > >> >>> // do gcd calculation > >> >>> } > >> >>> ``` > >> >>> > >> >>> We have a lot of boilerplate / non idea macro stuff in DataFusion > that > >> I > >> >>> think this would help a lot. > >> >>> > >> >>> Andrew > >> >>> > >> >>> > >> >>> On Fri, Jun 28, 2024 at 3:08 PM Raphael Taylor-Davies > >> >>> <r.taylordav...@googlemail.com.invalid> wrote: > >> >>> > >> >>>> I wonder if the DataFusion project might be a more natural home for > >> this > >> >>>> functionality? UDFs are more of a query engine concept, whereas > >> arrow-rs is > >> >>>> more focused on purely physical execution? > >> >>>> > >> >>>> On 28 June 2024 19:41:39 BST, Runji Wang <wangrunji0...@163.com> > >> wrote: > >> >>>>> Hi Felipe, > >> >>>>> > >> >>>>> Vectorization will be applied whenever possible. When all input > and > >> >>>> output types of a function are primitive (int16, int32, int64, > >> float32, > >> >>>> float64) and do not involve any Option or Result, the macro will > >> >>>> automatically generate code based on unary < > >> >>>> https://docs.rs/arrow/latest/arrow/compute/fn.unary.html> or > binary < > >> >>>> https://docs.rs/arrow/latest/arrow/compute/fn.binary.html> > kernels, > >> >>>> which potentially allows for vectorization. > >> >>>>> > >> >>>>> Both examples you showed are not vectorized. The `div` function is > >> due > >> >>>> to the Result output, while `gcd` is due to the loop in its > >> implementation. > >> >>>> However, if the function is simple enough, like an `add` function: > >> >>>>> > >> >>>>> #[function("add(int, int) -> int")] > >> >>>>> fn add(a: i32, b: i32) -> i32 { > >> >>>>> a + b > >> >>>>> } > >> >>>>> > >> >>>>> It can be auto-vectorized by llvm. > >> >>>>> > >> >>>>> Runji > >> >>>>> > >> >>>>> > >> >>>>> On 2024/06/28 17:13:16 Felipe Oliveira Carvalho wrote: > >> >>>>>> On Fri, Jun 28, 2024 at 11:07 AM Andrew Lamb < > al...@influxdata.com> > >> >>>> wrote: > >> >>>>>>> > >> >>>>>>> Hi Xuanwo, > >> >>>>>>> > >> >>>>>>> Sorry for the delay in responding. I think the ability to > easily > >> >>>> write > >> >>>>>>> functions that "feel" like native functions in whatever language > >> and > >> >>>> be > >> >>>>>>> able to generate arrow / vectorized versions of them is quite > >> >>>> valuable. > >> >>>>>>> This is my understanding of what this proposal is about. > >> >>>>>> > >> >>>>>> My understanding is that it's not vectorized. From the examples > in > >> >>>>>> risingwavelabs/arrow-udf, < > >> https://github.com/risingwavelabs/arrow-udf> > >> >>>> it > >> >>>>>> looks like the macros generate code that gathers values from > columns > >> >>>> into > >> >>>>>> local scalars that are passed as scalar parameters to user > >> functions. > >> >>>> Is > >> >>>>>> the hope here that rustc/llvm will auto-vectorize the code? > >> >>>>>> > >> >>>>>> #[function("gcd(int, int) -> int")] > >> >>>>>> fn gcd(mut a: i32, mut b: i32) -> i32 { > >> >>>>>> while b != 0 { > >> >>>>>> (a, b) = (b, a % b); > >> >>>>>> } > >> >>>>>> a > >> >>>>>> } > >> >>>>>> > >> >>>>>> #[function("div(int, int) -> int")] > >> >>>>>> fn div(x: i32, y: i32) -> Result<i32, &'static str> { > >> >>>>>> if y == 0 { > >> >>>>>> return Err("division by zero"); > >> >>>>>> } > >> >>>>>> Ok(x / y) > >> >>>>>> } > >> >>>>>> > >> >>>>>>> I left some additional comments on the markdown. > >> >>>>>>> > >> >>>>>>> One thing that might be worth doing is articulate some other > >> >>>> potential > >> >>>>>>> locations for where the code might go. One option, as I think > you > >> >>>> propose, > >> >>>>>>> is to make its own repository. Another option could be to > donate > >> >>>> the code > >> >>>>>>> and put the various language bindings in the same repo as the > arrow > >> >>>>>>> language implementations (e.g arrow-rs, arrow for python, etc) > >> which > >> >>>> would > >> >>>>>>> likely make it easier to maintain and discover. > >> >>>>>>> > >> >>>>>>> I am curious about what other devs / users feel about this? > >> >>>>>>> > >> >>>>>>> Andrew > >> >>>>>>> > >> >>>>>>> > >> >>>>>>> > >> >>>>>>> On Thu, Jun 20, 2024 at 3:04 AM Xuanwo <xu...@apache.org> > wrote: > >> >>>>>>> > >> >>>>>>>> Hello, everyone. > >> >>>>>>>> > >> >>>>>>>> I start this thread to disscuss the donation of a User-Defined > >> >>>> Function > >> >>>>>>>> Framework for Apache Arrow. > >> >>>>>>>> > >> >>>>>>>> Feel free to review and leave your comments here. For live > review, > >> >>>>>> please > >> >>>>>>>> visit: > >> >>>>>>>> > >> >>>>>>>> https://hackmd.io/@xuanwo/apache-arrow-udf > >> >>>>>>>> > >> >>>>>>>> The original content also pasted here for a quick reading: > >> >>>>>>>> > >> >>>>>>>> ------ > >> >>>>>>>> > >> >>>>>>>> ## Abstract > >> >>>>>>>> > >> >>>>>>>> Arrow UDF is a User-Defined Function Framework for Apache > Arrow. > >> >>>>>>>> > >> >>>>>>>> ## Proposal > >> >>>>>>>> > >> >>>>>>>> Arrow UDF allows user to easily create and run user-defined > >> >>>> functions > >> >>>>>>>> (UDF) in Rust, Python, Java or JavaScript based on Apache > Arrow. > >> >>>> The > >> >>>>>>>> functions can be executed natively, or in WebAssembly, or in a > >> >>>> remote > >> >>>>>>>> server via Arrow Flight. > >> >>>>>>>> > >> >>>>>>>> Arrow UDF was originally designed to be used by the RisingWave > >> >>>> project > >> >>>>>> but > >> >>>>>>>> is now being used by Databend and several database startups. > >> >>>>>>>> > >> >>>>>>>> We believe that the Arrow UDF project will provide diversity > value > >> >>>> to > >> >>>>>> the > >> >>>>>>>> entire Arrow community. > >> >>>>>>>> > >> >>>>>>>> ## Background > >> >>>>>>>> > >> >>>>>>>> Arrow UDF is being developed by an open-source community from > day > >> >>>> one > >> >>>>>> and > >> >>>>>>>> is owned by RisingWaveLabs. The project has been launched in > >> >>>> December > >> >>>>>> 2023. > >> >>>>>>>> > >> >>>>>>>> ## Initial Goals > >> >>>>>>>> > >> >>>>>>>> By transferring ownership of the project to the Apache Arrow, > >> >>>> Arrow UDF > >> >>>>>>>> expects to ensure its neutrality and further encourage and > >> >>>> facilitate > >> >>>>>> the > >> >>>>>>>> adoption of Arrow UDF by the community. > >> >>>>>>>> > >> >>>>>>>> ## Current Status > >> >>>>>>>> > >> >>>>>>>> Contributors: 5 > >> >>>>>>>> > >> >>>>>>>> Users: > >> >>>>>>>> > >> >>>>>>>> - [RisingWave]: A Distributed SQL Database for Stream > >> Processing. > >> >>>>>>>> - [Databend]: An open-source cloud data warehouse that > serves as > >> >>>> a > >> >>>>>>>> cost-effective alternative to Snowflake. > >> >>>>>>>> > >> >>>>>>>> ## Documentation > >> >>>>>>>> > >> >>>>>>>> The document of Arrow UDF is hosted at > >> >>>>>>>> https://docs.rs/arrow-udf/latest/arrow_udf/. > >> >>>>>>>> > >> >>>>>>>> ## Initial Source > >> >>>>>>>> > >> >>>>>>>> The project currently holds a GitHub repository and multiple > >> >>>> packages: > >> >>>>>>>> > >> >>>>>>>> - https://github.com/risingwavelabs/arrow-udf > >> >>>>>>>> > >> >>>>>>>> Rust: > >> >>>>>>>> > >> >>>>>>>> - https://crates.io/arrow-udf/ > >> >>>>>>>> - https://crates.io/arrow-udf-python/ > >> >>>>>>>> - https://crates.io/arrow-udf-js/ > >> >>>>>>>> - https://crates.io/arrow-udf-js-deno/ > >> >>>>>>>> - https://crates.io/arrow-udf-wasm/ > >> >>>>>>>> > >> >>>>>>>> Python: > >> >>>>>>>> > >> >>>>>>>> - https://pypi.org/project/arrow-udf/ > >> >>>>>>>> > >> >>>>>>>> Those packge will retain its name, while the repository will be > >> >>>> moved to > >> >>>>>>>> apache org. > >> >>>>>>>> > >> >>>>>>>> ## Required Resources > >> >>>>>>>> > >> >>>>>>>> ### Mailing Lists > >> >>>>>>>> > >> >>>>>>>> We can reuse the existing mailing lists that arrow have. > >> >>>>>>>> > >> >>>>>>>> ### Git Repositories > >> >>>>>>>> > >> >>>>>>>> From > >> >>>>>>>> > >> >>>>>>>> - https://github.com/risingwavelabs/arrow-udf > >> >>>>>>>> > >> >>>>>>>> To > >> >>>>>>>> > >> >>>>>>>> - https://gitbox.apache.org/asf/repos/arrow-udf > >> >>>>>>>> - https://github.com/apache/arrow-udf > >> >>>>>>>> > >> >>>>>>>> ### Issue Tracking > >> >>>>>>>> > >> >>>>>>>> The project would like to continue using GitHub Issues. > >> >>>>>>>> > >> >>>>>>>> ### Other Resources > >> >>>>>>>> > >> >>>>>>>> The project has already chosen GitHub actions as continuous > >> >>>> integration > >> >>>>>>>> tools. > >> >>>>>>>> > >> >>>>>>>> ## Initial Committers > >> >>>>>>>> > >> >>>>>>>> - Runji Wang wangrunji0...@163.com > >> >>>>>>>> - Giovanny Gutiérrez > >> >>>>>>>> - sundy-li sund...@apache.org > >> >>>>>>>> - Xuanwo xua...@apache.org > >> >>>>>>>> - Max Justus Spransy maxjus...@gmail.com > >> >>>>>>>> > >> >>>>>>>> [RisingWave]: https://github.com/risingwavelabs/risingwave > >> >>>>>>>> [Databend]: https://github.com/datafuselabs/databend > >> >>>>>>>> > >> >>>>>>>> Xuanwo > >> >>>>>>>> > >> >>>>>> > >> >>> > >> >>> > >> >> > >> > >> -- > >> Xuanwo > >> > > -- > Xuanwo >
