Re: I am using pyarrow 6.0.1 getting this error

2022-04-04 Thread Micah Kornfield
This type of error usually occurs when your environment has more than one version of pyarrow installed. On Mon, Apr 4, 2022 at 2:16 PM Uma Mudumba wrote: > "AttributeError: module 'pyarrow' has no attribute 'dataset'" at File > "parquet.py", line 1635 > > > can you please advise in resolving

I am using pyarrow 6.0.1 getting this error

2022-04-04 Thread Uma Mudumba
"AttributeError: module 'pyarrow' has no attribute 'dataset'" at File "parquet.py", line 1635 can you please advise in resolving this. thanks, Uma-

Re: PyArrow / Arrow questions about the time and date types

2022-04-04 Thread Wes McKinney
On Fri, Apr 1, 2022 at 2:00 PM Weston Pace wrote: > > > *Question 1*: For my own understanding: what purpose does the > > millisecond date64 type serve? > > I don't actually know the answer to this one. The rationale IIRC was that some systems represent dates this way, and so the purpose was to

Re: [Flight][Java][JDBC] IP clearance of Flight JDBC Driver

2022-04-04 Thread Wes McKinney
A corporate CLA is not required. Individual CLAs are fine. Since Dremio is a US corporation and the IP for the JDBC driver is owned by Dremio (I assume that the contributors all have IP assignment agreements where their contributions are assigned to the corporation), it would be best to have a

[RESULT] [VOTE][RUST] Release Apache Arrow Rust 11.1.0 RC1

2022-04-04 Thread Andrew Lamb
Thank you all very much 珞 The release is approved with 6 +1 votes (3 binding). It can be found at https://dist.apache.org/repos/dist/release/arrow/arrow-rs-11.1.0 It has also been uploaded to crates.io: https://crates.io/crates/arrow/11.1.0 https://crates.io/crates/arrow-flight/11.1.0

Re: [Question] Is it possible to write to IPC without an intermediary buffer?

2022-04-04 Thread Micah Kornfield
Hi Jorge, I don't think any implementation does this but I think it is technically possible, although it might be complicated to actually do. It also requires random access files (the output can't be purely streaming). I think the approach you would need to take is to pr-write the header

[Question] Is it possible to write to IPC without an intermediary buffer?

2022-04-04 Thread Jorge Cardoso Leitão
Hi, Motivated by [1], I wonder if it is possible to write to IPC without writing the data to an intermediary buffer. The challenge is that the header of an IPC message [header][data] requires: * the positions of the buffers * the total length of the body For uncompressed data, we could compute

Re: [C++] Replacing xsimd with compiler autovectorization

2022-04-04 Thread Antoine Pitrou
Le 03/04/2022 à 21:38, Sasha Krassovsky a écrit : There is concrete proof that autovectorization produces very flimsy results (even on the same compiler, simply by varying the datatypes). As I’ve shown, the Vector-Vector Add kernel example is consistently vectorized well across compilers