Re: testing of back-pressure

2023-02-16 Thread Li Jin
Thanks Weston for the information. On Thu, Feb 16, 2023 at 1:32 PM Weston Pace wrote: > There is a little bit at the end-to-end level. One goal is to be able to > repartition a very large dataset. This means we read from something bigger > than memory and then write to it. This workflow is te

Re: [DISCUSS] Flight RPC/Flight SQL/ADBC enhancements

2023-02-16 Thread Matthew Topol
While implementing Transaction handling for ADBC via Flight SQL's transaction primitives, another potential enhancement would be to expand the BeginTransaction request to include a spot for "options" such as IsolationLevel or marking a transaction as ReadOnly. Anyone have thoughts on this? On Wed

Re: [VOTE] Release Apache Arrow ADBC 0.2.0 - RC1

2023-02-16 Thread David Li
Post-release tasks: [x] Close the GitHub milestone/project [x] Add the new release to the Apache Reporter System [x] Upload source release artifacts to Subversion [x] Create the final GitHub release [x] Update website [x] Upload wheels/sdist to PyPI [x] Publish Maven packages [x] Update tags for G

[ANNOUNCE] Becoming a committer section added to website

2023-02-16 Thread Andrew Lamb
As part of our ongoing commitment to transparency and open communication, the PMC has written a summary of what is considered when evaluating new committers. You can find the write up under the "Becoming a committer" heading of [1] as well as the PR [2]. We hope this helps to clarify any uncertai

Re: testing of back-pressure

2023-02-16 Thread Weston Pace
There is a little bit at the end-to-end level. One goal is to be able to repartition a very large dataset. This means we read from something bigger than memory and then write to it. This workflow is tested in `test_write_dataset_with_backpresure` in test_dataset.py in pyarrow. Then there is a o

Re: Getting issues in cpp build

2023-02-16 Thread Shaheer Ahmad
There was a typo in my previous reply. I did not* specify CXX_STANDARD anywhere. On Thu, Feb 16, 2023 at 10:03 PM Shaheer Ahmad wrote: > I am giving the “cmake –build .” command within “arrow/cpp” directory. I > did specify CXX_STANDARD anywhere. Could you please guide me on that? > > On Thu,

Re: Getting issues in cpp build

2023-02-16 Thread Jacob Wujciak
Could it be an issue with an old version of c++.exe not supporting c++17? Shaheer could you also post the cmake prelude where it displays compiler version etc.? Shaheer Ahmad schrieb am Do., 16. Feb. 2023, 18:04: > I am giving the “cmake –build .” command within “arrow/cpp” directory. I > did

Re: Getting issues in cpp build

2023-02-16 Thread Shaheer Ahmad
I am giving the “cmake –build .” command within “arrow/cpp” directory. I did specify CXX_STANDARD anywhere. Could you please guide me on that? On Thu, Feb 16, 2023 at 9:59 PM Weston Pace wrote: > It appears you are tryingt o build with `-std=c++1z`. Arrow-C++ requires > C++17 (the header is

Re: Getting issues in cpp build

2023-02-16 Thread Weston Pace
It appears you are tryingt o build with `-std=c++1z`. Arrow-C++ requires C++17 (the header is only present in C++17 and greater). What command are you using to run cmake? Are you specifying CXX_STANDARD anywhere? On Wed, Feb 15, 2023 at 6:18 PM Shaheer Ahmad wrote: > [1/249] Building CXX obje

Re: Question about memory usage and type casting using pyarrow Table

2023-02-16 Thread Weston Pace
> (1) if I want to cast n columns to a different type (e.g., float to int). What is the smallest memory overhead that I can do? (memory overhead of 1 column, n columns or 100 columns?) You should be able to do this with only 1 column of overhead. Though you might need to go a little out of your w

testing of back-pressure

2023-02-16 Thread Yaron Gvili
Hi, What testing of back-pressure exist in Acero? I'm mostly interested in testing of back-pressure that applies to any ExecNode, but could also learn from more specific testing. If this is not well covered, I'd look into implementing such testing. Cheers, Yaron.

Re: [DISC] Self-Hosted Runners for Arrow

2023-02-16 Thread Martin Grigorov
Hi, On Tue, Feb 14, 2023 at 12:51 PM Raúl Cumplido wrote: > Hi, > > Following up on this thread, I am going to try and coordinate to set up an > instance of the self-hosted runners for arm64 on the Arrow repository. > > There was a question about using Travis CI on Crossbow for those jobs. That

Re: Proposal: renaming the 'master' branch to 'main'

2023-02-16 Thread Andy Grove
https://github.com/apache/arrow-datafusion default branch is now "main". Andy. On Tue, Feb 14, 2023 at 7:17 AM Andy Grove wrote: > I would like to rename the default branch in arrow-datafusion next. I have > a PR up with the required changes: > > https://github.com/apache/arrow-datafusion/pull/

Re: [VOTE] Release Apache Arrow ADBC 0.2.0 - RC1

2023-02-16 Thread Joris Van den Bossche
On Wed, 15 Feb 2023 at 21:31, Sutou Kouhei wrote: > > > not finding /usr/bin/mkdir > > Could you show the log of this? Yes: current directory: /tmp/adbc-verification/apache-arrow-adbc-0.2.0/glib/vendor/bundle/ruby/3.1.0/gems/fiddle-1.1.1/ext/fiddle make DESTDIR\= install make: /usr/bin/mkdir: Co