Re: [VOTE][RUST][DataFusion] Release Apache Arrow DataFusion 11.0.0 RC1

2022-08-17 Thread Martin Grigorov
+1 (Non-binding) Veirified on Ubuntu 20.04.4 x86_64 On Tue, Aug 16, 2022 at 7:16 PM Andy Grove wrote: > Hi, > > I would like to propose a release of Apache Arrow DataFusion > Implementation, > version 11.0.0. > > This release candidate is based on commit: > 8ee31cc69f43a4de0c0678d18a57f27cb4d0e

Re: [VOTE][RUST][DataFusion] Release Apache Arrow DataFusion 11.0.0 RC1

2022-08-17 Thread Remzi Yang
+1 (non-binding) Verified on M1 Mac. Thank you, Andy! On Thu, 18 Aug 2022 at 00:26, Francis Du wrote: > +1 (Non-binding) > > Verified on MacOS 12.5 / MBA M2 > > On Wed, 17 Aug 2022 at 16:47, Andrew Lamb wrote: > > > +1 (binding) > > > > Thanks Andy and everyone else who contributed to this rele

Re: [VOTE][RUST][DataFusion] Release Apache Arrow DataFusion 11.0.0 RC1

2022-08-17 Thread Remzi Yang
Sorry, my mistake, I forgot updating the verification script. On Wed, 17 Aug 2022 at 16:46, Andrew Lamb wrote: > This looks similar to [1] > > Do you by any chance have the ARROW_TEST_DATA environment set? If so I > think it needs to end with a `/` or be unset to run the script. > > The differen

Re: [VOTE][RUST][DataFusion] Release Apache Arrow DataFusion 11.0.0 RC1

2022-08-17 Thread QP Hou
+1 (binding) On Wed, Aug 17, 2022 at 9:26 AM Francis Du wrote: > +1 (Non-binding) > > Verified on MacOS 12.5 / MBA M2 > > On Wed, 17 Aug 2022 at 16:47, Andrew Lamb wrote: > > > +1 (binding) > > > > Thanks Andy and everyone else who contributed to this release -- this > looks > > like it will be

Re: DISCUSS: [C++] Switch to C++17

2022-08-17 Thread Weston Pace
> Any particular reason why this should be 10.0 and not 9.0 for example? (is due to an incoming feature of note?) No. I only said 10.0 because Neal's tactical suggestion earlier in this thread would mean that 10.0 would be the last build that had C++11 support. If we choose not to follow that sug

Re: Arrow sync call August 17 at 12:00 US/Eastern, 16:00 UTC

2022-08-17 Thread Rok Mihevc
Attendees: Matt Topol Will Jones David Li Joris Van den Bosche Eduardo Ponce Atoine Pitrou Jacob Wujciak Ivan Ogasawara Ashish Paliwal Niranda Perera Discussion: - FlightSQL PR reviews David Li is inviting reviewers to two FlightSQL PRs. [1] "ARROW-7744: [Java][FlightRPC] JDBC Driver for Arrow F

Re: DISCUSS: [C++] Switch to C++17

2022-08-17 Thread Sutou Kouhei
Hi, > Here's a tactical suggestion: for 10.0, we could upgrade all* of our > packaging and CI jobs to build with C++17, but not add code that cannot > compile on C++11 (hence the *, we will need to maintain at least one C++11 > CI job to assure that). Then, assuming there are no adverse consequenc

Re: [C++] Read Flight data source into Acero

2022-08-17 Thread David Li
Convert the reader to a RecordBatchReader [1], turn it into an AsyncGenerator [2], then wrap it in a SourceNode (though I think SourceNode should just expose a helper for wrapping a RecordBatchReader like it already does for Tables). [1]: https://github.com/apache/arrow/blob/8474ee5a3ed725d4bb5

Re: [C++] Read Flight data source into Acero

2022-08-17 Thread Aldrin
I don't have any pointers, but just wanted to mention that I am going to try and figure this out quite a bit in the next week. I can try to create some relevant cookbook recipes as I plod along. Aldrin Montana Computer Science PhD Student UC Santa Cruz On Wed, Aug 17, 2022 at 9:15 AM Li Jin wro

Re: DISCUSS: [Format] Rules and procedures for Canonical extension types

2022-08-17 Thread Joris Van den Bossche
+1 on the overall proposal, documenting those in a central place sounds good to me. On Wed, 17 Aug 2022 at 18:10, Antoine Pitrou wrote: > > > > * The specification text to be added *must* follow these requirements > > 1) It *must* have a well-defined name starting with "ARROW:" > One remar

Re: [VOTE][RUST][DataFusion] Release Apache Arrow DataFusion 11.0.0 RC1

2022-08-17 Thread Francis Du
+1 (Non-binding) Verified on MacOS 12.5 / MBA M2 On Wed, 17 Aug 2022 at 16:47, Andrew Lamb wrote: > +1 (binding) > > Thanks Andy and everyone else who contributed to this release -- this looks > like it will be great! > > On Tue, Aug 16, 2022 at 1:17 PM Ian Joiner wrote: > > > +1 (Non-binding)

Re: [C++] Read Flight data source into Acero

2022-08-17 Thread Li Jin
Correction: I have a flight::FlightStreamReader (not Flight::StreamReader) On Wed, Aug 17, 2022 at 12:12 PM Li Jin wrote: > Hi, > > I have a Flight data source (effectively a Flight::StreamReader) and I'd > like to create an Acero source node from it. I wonder if something already > exists to do

[C++] Read Flight data source into Acero

2022-08-17 Thread Li Jin
Hi, I have a Flight data source (effectively a Flight::StreamReader) and I'd like to create an Acero source node from it. I wonder if something already exists to do that or if not, perhaps some pointers for me to take a look at? Thanks, Li

DISCUSS: [Format] Rules and procedures for Canonical extension types

2022-08-17 Thread Antoine Pitrou
Hello all, The Arrow format has support for extension types, but there's no official way to agree accross implementations on well-known extension types. This issue has come up a couple times with people wanting to implement support for types such as JSON or UUID in order to enable better i

Re: DISCUSS: [C++] Switch to C++17

2022-08-17 Thread Keith Kraus
+1 (non-binding) >From having previously run a large C++ project that migrated from C++11 to C++17, there was a huge quality of life improvement for developers and it made attracting new developers much easier. One potential pitfall, C++17 wasn't supported by NVIDIA compilers until CUDA Toolkit 1

Re: DISCUSS: [C++] Switch to C++17

2022-08-17 Thread Jacob Wujciak
+ 1 After Antoine implemented a work around for an issue with optional::value [1][2], we were able to successfully build cpp standalone (cpp_build.sh) and the R package [3] on the 10.13 runner that is setup to match the CRAN builder. (The run has an R test failure but that is most likely unrelated)

Re: [ARROW-17255] Logical JSON type in Arrow

2022-08-17 Thread Antoine Pitrou
Hi Pradeep, Thanks for filing this PR! Before merging this PR, I think we should discuss a bit what a canonical extension type is, and how it gets standardized. I'll make a separate discussion thread. Regards Antoine. Le 16/08/2022 à 22:40, Pradeep Gollakota a écrit : Hi all, I've cre

Re: [C++] Can we drop support for Visual Studio 2017?

2022-08-17 Thread Antoine Pitrou
No opposition from me. Regards Antoine. Le 17/08/2022 à 10:05, Sutou Kouhei a écrit : Hi, Can we drop support for Visual Studio 2017? Visual Studio 2017 reached EOL at 2022-04-12: https://docs.microsoft.com/en-us/lifecycle/products/visual-studio-2017 Listing| Start Date | Ma

Re: DISCUSS: [C++] Switch to C++17

2022-08-17 Thread Antoine Pitrou
Le 17/08/2022 à 16:52, Weston Pace a écrit : Sorry for a "one more thing email" but I had one more thought regarding R 3.6 support for Windows. I think those users should continue to be able to use Arrow 10.0.0. Any particular reason why this should be 10.0 and not 9.0 for example? (is due t

Re: DISCUSS: [C++] Switch to C++17

2022-08-17 Thread Weston Pace
Sorry for a "one more thing email" but I had one more thought regarding R 3.6 support for Windows. I think those users should continue to be able to use Arrow 10.0.0. So this would only be an issue if we detected a critical bug in 10.0.0 (which we could backport if it was bad enough) or if they w

Re: Proposal: Allow any ASF Jira user to assign ARROW issues

2022-08-17 Thread Todd Farmer
Thanks for the feedback - I've created the following to request changes to the ARROW project issue assignment scheme: https://issues.apache.org/jira/browse/INFRA-23589 Duncan: It's particularly helpful to get your feedback on friction you experience as a recent first-time contributor. I'd very mu

Re: dealing with tester timeout in a CI job

2022-08-17 Thread Weston Pace
Yes, sorry, SCOPED_TRACE and not SCOPED_TEST (and ARROW_SCOPED_TRACE, which hides SCOPED_TRACE, is best). You can read about it at [1] but it basically says "Here is a message that should be printed if the test fails in this block of code" [1] http://google.github.io/googletest/advanced.html#addi

Re: DISCUSS: [C++] Switch to C++17

2022-08-17 Thread Weston Pace
+1. I'm very much in favor of upgrading to C++17. I am lucky to often get to work with people that are new to the Arrow C++ code base and a common feedback is that the code is quite complex. While I do not think moving to C++17 will solve this problem by itself I'm pretty confident that being ab

Re: DISCUSS: [C++] Switch to C++17

2022-08-17 Thread Matthew Topol
I can definitely vouch for the benefits of upgrading to C++17, it makes general development significantly more enjoyable! On that end, I second Neal's suggestion for deployment, that's roughly the same plan we used in my past experience when performing a compiler upgrade on a very large codeba

Re: dealing with tester timeout in a CI job

2022-08-17 Thread Antoine Pitrou
Look for ARROW_SCOPED_TRACE Le 17/08/2022 à 16:22, Yaron Gvili a écrit : There are no sleeps nor deadlocks; it's just due to a large configuration-space that I agree can be reduced by sampling. Could you explain how to use SCOPED_TEST, or refer to documentation about it? I understand your id

Re: dealing with tester timeout in a CI job

2022-08-17 Thread Yaron Gvili
There are no sleeps nor deadlocks; it's just due to a large configuration-space that I agree can be reduced by sampling. Could you explain how to use SCOPED_TEST, or refer to documentation about it? I understand your idea, just looking for an example use of SCOPED_TEST. Yaron.

Re: dealing with tester timeout in a CI job

2022-08-17 Thread Weston Pace
My first suspicion on a test timeout is usually a deadlock. That being said, I haven't looked at this test / change in any real detail so I don't know if that's the case here. How long does the test take to run locally? Second, I would try and remove sleeps, and make sure to use the utilities Sl

Re: DISCUSS: [C++] Switch to C++17

2022-08-17 Thread Neal Richardson
I have no opinion about the benefits of upgrading to C++17. From the R perspective, there are a handful of packages on CRAN that require C++14 or C++17. Last year, when I asked other R package maintainers why they hadn't upgraded to newer C++ standards, the reasoning was that because many packages

Re: dealing with tester timeout in a CI job

2022-08-17 Thread Yaron Gvili
It looks like the test normally takes less than a second. The gap in running-time is not surprising because the tests I locally added cover a much larger configuration-space. Before I reduce the configuration-space being tested, I'd like to figure out what the acceptable alternatives are. Yaro

Re: dealing with tester timeout in a CI job

2022-08-17 Thread Li Jin
Yaron, how does the asof join tests normally take? On Wed, Aug 17, 2022 at 6:13 AM Yaron Gvili wrote: > Sorry, yes, C++. The failed job is > https://github.com/apache/arrow/runs/7839062613?check_suite_focus=true > and it timed out on code I wrote (in a PR, not merged). I'd like to avoid a > time

Arrow sync call August 17 at 12:00 US/Eastern, 16:00 UTC

2022-08-17 Thread Ian Cook
Hi all, Our biweekly sync call is today at 12:00 noon Eastern time. The Zoom meeting URL for this and other biweekly Arrow sync calls is: https://zoom.us/j/87649033008?pwd=SitsRHluQStlREM0TjJVYkRibVZsUT09 Alternatively, enter this information into the Zoom website or app to join the call: Meetin

Re: dealing with tester timeout in a CI job

2022-08-17 Thread Yaron Gvili
Sorry, yes, C++. The failed job is https://github.com/apache/arrow/runs/7839062613?check_suite_focus=true and it timed out on code I wrote (in a PR, not merged). I'd like to avoid a timeout without reengineering or reducing the set of tests I wrote, hence my questions. Yaron. _

Re: DISCUSS: [C++] Switch to C++17

2022-08-17 Thread Antoine Pitrou
Le 17/08/2022 à 10:48, Jacob Wujciak a écrit : I am generally in favour of this proposal but would like to mention that we have to be able to build on MacOS 10.13 for the R package due to CRAN using it. The CRAN builder comes with: Apple LLVM version 10.0.0 (clang-1000.10.44.4); GNU Fortran (GC

Re: DISCUSS: [C++] Switch to C++17

2022-08-17 Thread Jacob Wujciak
I am generally in favour of this proposal but would like to mention that we have to be able to build on MacOS 10.13 for the R package due to CRAN using it. The CRAN builder comes with: Apple LLVM version 10.0.0 (clang-1000.10.44.4); GNU Fortran (GCC) 8.2.0 Antoine Pitrou schrieb am Mi., 17. Aug.

Re: [VOTE][RUST][DataFusion] Release Apache Arrow DataFusion 11.0.0 RC1

2022-08-17 Thread Andrew Lamb
+1 (binding) Thanks Andy and everyone else who contributed to this release -- this looks like it will be great! On Tue, Aug 16, 2022 at 1:17 PM Ian Joiner wrote: > +1 (Non-binding) > > Verified on macOS 12.2.1 / Apple M1 Pro > > P.S. If verified with zsh instead of bash we got a command not fou

Re: [VOTE][RUST][DataFusion] Release Apache Arrow DataFusion 11.0.0 RC1

2022-08-17 Thread Andrew Lamb
This looks similar to [1] Do you by any chance have the ARROW_TEST_DATA environment set? If so I think it needs to end with a `/` or be unset to run the script. The difference is there is something wrong with the normalization: Expected: > files=[ARROW_TEST_DATA/csv/aggregate_test_100.csv], has_

Re: [C++] Moving from -O3 to -O2 optimization level in release builds

2022-08-17 Thread Antoine Pitrou
For the record, https://github.com/apache/arrow/pull/13661 was finally merged. It switches to -O2 by default and selectively re-enables auto-vectorization on gcc. Regards Antoine. Le 21/07/2022 à 17:11, Antoine Pitrou a écrit : Le 21/07/2022 à 16:34, Wes McKinney a écrit : Based on t

DISCUSS: [C++] Switch to C++17

2022-08-17 Thread Antoine Pitrou
Hello, We are in 2022 and Arrow C++ still strives to be compatible with C++11. Maintaining compatibility has caused us growing pains since third-party libraries have begun requiring C++14 or later. Boost is warning that it will soon require C++14 (https://issues.apache.org/jira/browse/ARROW

[C++] Can we drop support for Visual Studio 2017?

2022-08-17 Thread Sutou Kouhei
Hi, Can we drop support for Visual Studio 2017? Visual Studio 2017 reached EOL at 2022-04-12: https://docs.microsoft.com/en-us/lifecycle/products/visual-studio-2017 Listing| Start Date | Mainstream End Date | Extended End Date Visual Studio 2017 | Mar 7, 2017 | Apr 12, 2022|