Re: [DISCUSS] New Types (Schema.fbs vs Extension Types)

2021-04-29 Thread Jorge Cardoso Leitão
Thanks for writing this. I agree. That is a good decision tree. +1 Best, Jorge On Thu, Apr 29, 2021 at 6:08 PM Micah Kornfield wrote: > The discussion around adding another interval type to the Schema.fbs raises > the issue of when do we decide to add a new type to the Schema.fbs vs using >

Re: [ANNOUNCE] New Arrow committer: Jonathan Keane

2021-04-29 Thread Ying Zhou
Congrats Jonathan! > On Apr 28, 2021, at 5:20 PM, David Li wrote: > > Congrats Jonathan! > > -David > > On Wed, Apr 28, 2021, at 16:55, Jorge Cardoso Leitão wrote: >> Congratulations and thank you for your contributions :) >> >> On Wed, Apr 28, 2021 at 10:37 PM Neal Richardson < >>

Re: [ANNOUNCE] New Arrow committer: Ian Cook

2021-04-29 Thread Ying Zhou
Congrats Ian! > On Apr 28, 2021, at 7:01 PM, paddy horan wrote: > > Congrats Ian! > > > > From: Jorge Cardoso Leit?o > Sent: Wednesday, April 28, 2021 4:56:12 PM > To: dev@arrow.apache.org > Subject: Re: [ANNOUNCE] New Arrow committer: Ian Cook > >

[C++] Adopting a library for (distributed) tracing

2021-04-29 Thread David Li
Hello, For Arrow Datasets, I've been working to instrument the scanner to find bottlenecks. For example, here's a demo comparing the current async scanner, which doesn't truly read asynchronously, to one that does; it should be fairly evident where the bottleneck is:

Re: [Python] Reusable Flight Middleware

2021-04-29 Thread David Li
Thanks for the context. I'll be posting later today so I'd appreciate any feedback you have there as well. I agree with the caveat about bridging C++<->Python. I don't think this is accounted for upstream, IIRC the best you can do is ship a span context across the FFI boundary, and otherwise just

Re: [Python] Reusable Flight Middleware

2021-04-29 Thread Eric Henry
Thanks for the response David. That proposal sounds exactly what I'm looking for in terms of middleware. My motivation for tracing comes from previously working within a complex real time orchestration system with 20+ microservices involved in creating various features per request in the system.

Re: [Python] Reusable Flight Middleware

2021-04-29 Thread David Li
Hey Eric, It so happens that I am literally in the middle of drafting a proposal to the mailing list to integrate with OpenTelemetry on the C++ level. I would fully support having these capabilities for Flight, and I think in the past we've discussed having a contrib directory for such modules,

[Python] Reusable Flight Middleware

2021-04-29 Thread Eric Henry
Is there currently any interest or effort underway in providing reusable Flight middleware for observability and production usage (e.g. logging, tracing, metrics)? For my own usage and experimentation I created a very basic json logging and OpenTracing implementation, but I was wondering if there

Re: [ANNOUNCE] New Arrow committer: Jonathan Keane

2021-04-29 Thread Micah Kornfield
Congrats! On Thu, Apr 29, 2021 at 10:35 AM Rok Mihevc wrote: > Congrats Jonathan! > > On Thu, Apr 29, 2021 at 6:10 PM Mauricio Vargas < > mauri...@ursacomputing.com> > wrote: > > > congrats !! > > > > On Thu, Apr 29, 2021 at 3:47 AM Daniël Heres > > wrote: > > > > > Congrats! > > > > > > Op wo

Re: [ANNOUNCE] New Arrow committer: Ian Cook

2021-04-29 Thread Micah Kornfield
Congrats! On Thu, Apr 29, 2021 at 10:35 AM Rok Mihevc wrote: > Congrats Ian! > > On Thu, Apr 29, 2021 at 9:46 AM Daniël Heres > wrote: > > > Congrats! > > > > Op do 29 apr. 2021 om 01:08 schreef paddy horan >: > > > > > Congrats Ian! > > > > > > > > > > > >

Re: [ANNOUNCE] New Arrow committer: Ian Cook

2021-04-29 Thread Rok Mihevc
Congrats Ian! On Thu, Apr 29, 2021 at 9:46 AM Daniël Heres wrote: > Congrats! > > Op do 29 apr. 2021 om 01:08 schreef paddy horan : > > > Congrats Ian! > > > > > > > > From: Jorge Cardoso Leit?o > > Sent: Wednesday, April 28, 2021 4:56:12 PM > > To:

Re: [ANNOUNCE] New Arrow committer: Jonathan Keane

2021-04-29 Thread Rok Mihevc
Congrats Jonathan! On Thu, Apr 29, 2021 at 6:10 PM Mauricio Vargas wrote: > congrats !! > > On Thu, Apr 29, 2021 at 3:47 AM Daniël Heres > wrote: > > > Congrats! > > > > Op wo 28 apr. 2021 om 23:21 schreef David Li : > > > > > Congrats Jonathan! > > > > > > -David > > > > > > On Wed, Apr 28,

Re: [ANNOUNCE] New Arrow committer: Jonathan Keane

2021-04-29 Thread Mauricio Vargas
congrats !! On Thu, Apr 29, 2021 at 3:47 AM Daniël Heres wrote: > Congrats! > > Op wo 28 apr. 2021 om 23:21 schreef David Li : > > > Congrats Jonathan! > > > > -David > > > > On Wed, Apr 28, 2021, at 16:55, Jorge Cardoso Leitão wrote: > > > Congratulations and thank you for your contributions

[DISCUSS] New Types (Schema.fbs vs Extension Types)

2021-04-29 Thread Micah Kornfield
The discussion around adding another interval type to the Schema.fbs raises the issue of when do we decide to add a new type to the Schema.fbs vs using other means (primarily extension types [1]). A few criteria come to mind that could help decide (feedback welcome): 1. Is the type a new

Re: Independent releases and format version

2021-04-29 Thread Wes McKinney
When in doubt, return to the format versioning policy: https://arrow.apache.org/docs/format/Versioning.html The interval type addition is a MINOR version increase in the format version. As a result of the forward compatibility policy, old readers are expected to be able to recognize when they

[NIGHTLY] Arrow Build Report for Job nightly-2021-04-29-0

2021-04-29 Thread Crossbow
Arrow Build Report for Job nightly-2021-04-29-0 All tasks: https://github.com/ursacomputing/crossbow/branches/all?query=nightly-2021-04-29-0 Failed Tasks: - conda-linux-gcc-py36-arm64: URL:

Re: Independent releases and format version

2021-04-29 Thread Antoine Pitrou
Le 29/04/2021 à 02:26, Weston Pace a écrit : There is also a potential format change coming up (new interval type). Ok, so more accurately, it is not a format change, it's a format addition ;-) This sounds pedantic but a format change would potentially break compatibility (for example

Re: Independent releases and format version

2021-04-29 Thread Antoine Pitrou
Le 29/04/2021 à 02:26, Weston Pace a écrit : We now have independent releases. There has been some discussion (not sure if it was formalized) around aligning major release versions across the languages. There is also a potential format change coming up (new interval type). I think this

Re: Independent releases and format version

2021-04-29 Thread Weston Pace
Ok, so it sounds less like a "library" dependency and more like the relationship between C++ specification and compiler or between browser features and browsers. Does an implementation have to implement the 1.0 spec before implementing features in the 1.1 spec? I would assume no. Given that we

Re: [ANNOUNCE] New Arrow committer: Ian Cook

2021-04-29 Thread Daniël Heres
Congrats! Op do 29 apr. 2021 om 01:08 schreef paddy horan : > Congrats Ian! > > > > From: Jorge Cardoso Leit?o > Sent: Wednesday, April 28, 2021 4:56:12 PM > To: dev@arrow.apache.org > Subject: Re: [ANNOUNCE] New Arrow committer: Ian Cook > > Congratulations

Re: [ANNOUNCE] New Arrow committer: Jonathan Keane

2021-04-29 Thread Daniël Heres
Congrats! Op wo 28 apr. 2021 om 23:21 schreef David Li : > Congrats Jonathan! > > -David > > On Wed, Apr 28, 2021, at 16:55, Jorge Cardoso Leitão wrote: > > Congratulations and thank you for your contributions :) > > > > On Wed, Apr 28, 2021 at 10:37 PM Neal Richardson < > >