Re: Flight/FlightSQL Optimization for Small Results?

2022-03-08 Thread Micah Kornfield
> > The operation flow would be like this, or what would it look like? > Client ---> GetFlightInfo (query/update operation in payload) ---> Server > ---> Results (non-streamed) This is roughly the flow I was imagining if the server chooses to send back inlined data. -Micah On Tue, Mar 8, 2022

Re: [Discuss][Format] Add 32-bit and 64-bit Decimals

2022-03-08 Thread Jorge Cardoso Leitão
Agreed. Also, I would like to revise my previous comment about the small risk. While prototyping this I did hit some bumps. They primary came from two reasons: * I was unable to find arrow/json files in the arrow-testing generated files with a non-default decimal bitwidth (I think we only have

Re: [DISCUSS][Rust] Biweekly sync call for arrow/datafusion again?

2022-03-08 Thread Andrew Lamb
I am not sure if everyone saw it in the agenda[1], but we plan to have a meeting tomorrow. I'll plan to record it for anyone who can not make this time. 15:00 UTC Wednesday March 9, 2022 Meeting Location: (in agenda) Matthew Turner: focused on JIT and row representation, next Wednesday, March

Re: [Discuss][Format] Add 32-bit and 64-bit Decimals

2022-03-08 Thread Micah Kornfield
> > I’d also like to chime in in favor of 32- and 64-bit decimals because > it’ll help achieve better performance on TPC-H (and maybe other > benchmarks). The decimal columns need only 12 digits of precision, for > which a 64-bit decimal is sufficient. It’s currently wasteful to use a > 128-bit

Re: Flight/FlightSQL Optimization for Small Results?

2022-03-08 Thread Gavin Ray
Thank you for doing this, left a few questions on the GH issue I would adopt this proposal as soon as it makes it into nightlies (or possibly earlier if it's just a matter of regenerating the proto definitions) The operation flow would be like this, or what would it look like? Client --->

Re: [Discuss][Format] Add 32-bit and 64-bit Decimals

2022-03-08 Thread Sasha Krassovsky
I’d also like to chime in in favor of 32- and 64-bit decimals because it’ll help achieve better performance on TPC-H (and maybe other benchmarks). The decimal columns need only 12 digits of precision, for which a 64-bit decimal is sufficient. It’s currently wasteful to use a 128-bit decimal.

Re: Flight/FlightSQL Optimization for Small Results?

2022-03-08 Thread Micah Kornfield
Some people have already left comments on https://github.com/apache/arrow/pull/12571 More eyes on it would be appreciated. If there aren't more comments, I'll try to start implementing this feature in Flight next week, and hopefully have a vote after it is supported in Java and C++/Python.

Re: [Discuss][Format] Add 32-bit and 64-bit Decimals

2022-03-08 Thread Micah Kornfield
> > Do we want to keep the historical "C++ and Java" requirement or > do we want to make it a more flexible "two independent official > implementations", which could be for example C++ and Rust, Rust and > Java, etc. I think flexibility here is a good idea, I'd like to hear other opinions. For

[RESULT][VOTE][RUST] Release Apache Arrow Rust 10.0.0 RC1

2022-03-08 Thread Andrew Lamb
With 8 +1 (3 binding) the release is approved! Thank you to all who verified it. The release is available here: https://dist.apache.org/repos/dist/release/arrow/arrow-rs-10.0.0 It has also been uploaded to crates.io: https://crates.io/crates/arrow/10.0.0

Re: [Discuss][Format] Add 32-bit and 64-bit Decimals

2022-03-08 Thread Antoine Pitrou
Le 07/03/2022 à 20:26, Micah Kornfield a écrit : Relaxing from {128,256} to {32,64,128,256} seems a low risk from an integration perspective, as implementations already need to read the bitwidth to select the appropriate physical representation (if they support it). I think there are two