Jorge created ARROW-9918: ---------------------------- Summary: [Rust] [DataFusion] Move away from builders to create arrays Key: ARROW-9918 URL: https://issues.apache.org/jira/browse/ARROW-9918 Project: Apache Arrow Issue Type: Improvement Components: Rust, Rust - DataFusion Reporter: Jorge Assignee: Jorge
[~yordan-pavlov] hinted in https://issues.apache.org/jira/browse/ARROW-8908 that there is a performance hit when using array builders. Indeed, I get about 15% speedup by using {{From}} instead of a builder in DataFusion's math operations, and this number _increases with batch size_, which is detrimental since larger batch sizes leverage SDMI better. Below, {{sqrt_X_Y}}, X is the log2 of the array's size, Y is log2 of the batch size: {code:java} sqrt_20_12 time: [34.422 ms 34.503 ms 34.584 ms] change: [-16.333% -16.055% -15.806%] (p = 0.00 < 0.05) Performance has improved. sqrt_22_12 time: [150.13 ms 150.79 ms 151.42 ms] change: [-16.281% -15.488% -14.779%] (p = 0.00 < 0.05) Performance has improved. sqrt_22_14 time: [151.45 ms 151.68 ms 151.90 ms] change: [-18.233% -16.919% -15.888%] (p = 0.00 < 0.05) Performance has improved. {code} See how there is no difference between {{sqrt_20_12}} and {{sqrt_22_12}} (same batch size), but a measurable difference between {{sqrt_22_12}} and {{sqrt_22_14}} (different batch size). This issue proposes that we migrate our datafusion and rust operations from builder-base array construction to a non-builder based, whenever possible. Most code can be easily migrated to use {{From}} trait, a migration is along the lines of {code:java} let mut builder = Float64Builder::new(array.len()); for i in 0..array.len() { if array.is_null(i) { builder.append_null()?; } else { builder.append_value(array.value(i).$FUNC())?; } } Ok(Arc::new(builder.finish())) {code} to {code:java} let result: Float64Array = (0..array.len()) .map(|i| { if array.is_null(i) { None } else { Some(array.value(i).$FUNC()) } }) .collect::<Vec<Option<f64>>>() .into(); Ok(Arc::new(result)){code} and this is even simpler as we do not need to use an extra API (Builder). -- This message was sent by Atlassian Jira (v8.3.4#803005)