Re: [VOTE] Release Apache Arrow 7.0.0 - RC10

2022-02-08 Thread Ian Joiner
Sure. Let’s actually keep it simple. Just go to https://arrow.apache.org/docs/index.html and click on “7.0.0”. The problem will immediately manifest itself. Ian On Wednesday, February 9, 2022, Sutou Kouhei wrote: > Thanks. > But we can't attach an image to this mailing list. Could you > upload

Re: [VOTE] Release Apache Arrow 7.0.0 - RC10

2022-02-08 Thread Sutou Kouhei
Thanks for confirming them! I've pressed the "Release" button on https://repository.apache.org/ . In "Re: [VOTE] Release Apache Arrow 7.0.0 - RC10" on Tue, 8 Feb 2022 21:06:18 -0800, Chao Sun wrote: > The Java Maven artifacts look good to me. I was able to upgrade Apache > Spark to use

Re: [VOTE] Release Apache Arrow 7.0.0 - RC10

2022-02-08 Thread Sutou Kouhei
Thanks. But we can't attach an image to this mailing list. Could you upload it to somewhere such as https://gist.github.com/ ? In "Re: [VOTE] Release Apache Arrow 7.0.0 - RC10" on Tue, 8 Feb 2022 20:51:16 -0500, Ian Joiner wrote: > The URLs are good. It is the values in the drop-down list

Re: [VOTE] Release Apache Arrow 7.0.0 - RC10

2022-02-08 Thread Chao Sun
The Java Maven artifacts look good to me. I was able to upgrade Apache Spark to use the staging artifacts and all the tests successfully passed. It's also great to see the `arrow-c-data` module finally get published in this release! Chao On Tue, Feb 8, 2022 at 5:51 PM Ian Joiner wrote: > The

Re: [Discuss] Best practice for storing key-value metadata for Extension Types

2022-02-08 Thread Dewey Dunnington
I'll share a bit more about geospatial extension types that Joris mentioned. I'm new to the Arrow community and didn't know that there were any restrictions on metadata values (the C Data interface docs don't seem to indicate that there are restrictions, or if it's there I missed it!), so I used

Re: [VOTE] Release Apache Arrow 7.0.0 - RC10

2022-02-08 Thread Ian Joiner
The URLs are good. It is the values in the drop-down list that still need to be fixed. Please see the attached photo. Ian On Tuesday, February 8, 2022, Sutou Kouhei wrote: > > Could you show URL that is occurred? > > It seems that the following URLs show correct versions: > > *

Re: [Discuss] Best practice for storing key-value metadata for Extension Types

2022-02-08 Thread Weston Pace
I think I'm +0 but lean slightly towards JSON. In favor of binary I would guess that most extension types are going to have relatively simple parameterization (to the point that protobuf/flatbuffers isn't really needed). For example, the substrate consumer PR has five extension types at the

Re: [VOTE] Release Apache Arrow 7.0.0 - RC10

2022-02-08 Thread Sutou Kouhei
Could you show URL that is occurred? It seems that the following URLs show correct versions: * https://arrow.apache.org/docs/6.0/index.html * https://arrow.apache.org/docs/index.html * https://arrow.apache.org/docs/dev/index.html Thanks, -- kou In "Re: [VOTE] Release Apache Arrow 7.0.0 -

Re: [VOTE] Release Apache Arrow 7.0.0 - RC10

2022-02-08 Thread Ian Joiner
Really thanks! I do need to mention that versioning in the docs is still not displayed properly (7.0 is labelled “6.0 (stable)” while 8.0 is labelled “7.0 (dev)”). Ian On Tuesday, February 8, 2022, Sutou Kouhei wrote: > Homebrew, MSYS2 and RubyGems are done: > > 1. [done] make the released

Re: [VOTE] Release Apache Arrow 7.0.0 - RC10

2022-02-08 Thread Chao Sun
I just opened a PR in Spark to test the Arrow upgrade: https://github.com/apache/spark/pull/35449. Let's see how it goes. > Technical question: can we rollback releases on maven? Hmm you mean remove already published artifacts? I don't think there is a way: once artifacts are published, they are

Re: [VOTE] Release Apache Arrow 7.0.0 - RC10

2022-02-08 Thread Krisztián Szűcs
On Tue, Feb 8, 2022 at 10:01 PM Chao Sun wrote: > > Thanks, let me verify the artifacts and come back to this thread. That would be great, thanks Chao! > > On Tue, Feb 8, 2022 at 1:00 PM Sutou Kouhei wrote: > > > Hi, > > > > > Just curious what's the progress of publishing Maven artifacts.

Re: [DISCUSS] Further proposals for Flight SQL

2022-02-08 Thread David Li
Thanks for the clarification, Wes! Kyle - the grant process is outlined here [1] and I can help with this on the Arrow PMC side. From your side, you will need to file a grant (either the CCLA form or the grant here [2]) and make sure everyone has a CLA on file, then once the Apache side has

Re: [VOTE] Release Apache Arrow 7.0.0 - RC10

2022-02-08 Thread Chao Sun
Thanks, let me verify the artifacts and come back to this thread. On Tue, Feb 8, 2022 at 1:00 PM Sutou Kouhei wrote: > Hi, > > > Just curious what's the progress of publishing Maven artifacts. Thanks! > > It seems that Krisztián is waiting for someone's > verification of the staged artifacts: >

Re: [VOTE] Release Apache Arrow 7.0.0 - RC10

2022-02-08 Thread Sutou Kouhei
Hi, > Just curious what's the progress of publishing Maven artifacts. Thanks! It seems that Krisztián is waiting for someone's verification of the staged artifacts: https://lists.apache.org/thread/r7cm1zz2qz9r823p5tbxdv0gtr3o6s4r > 14. [todo] publish Maven artifacts > Micah did you have a

Re: [VOTE] Release Apache Arrow 7.0.0 - RC10

2022-02-08 Thread Sutou Kouhei
Homebrew, MSYS2 and RubyGems are done: 1. [done] make the released version as "RELEASED" on JIRA 2. [done] start the new version on JIRA 4. [done] upload source 5. [done] upload binaries 6. [done] update website 7. [done] update Homebrew packages 8. [done] update MSYS2 package 9. [done] upload

Re: [VOTE] Release Apache Arrow 7.0.0 - RC10

2022-02-08 Thread Chao Sun
Hi Krisztián, Just curious what's the progress of publishing Maven artifacts. Thanks! Best, Chao On Thu, Feb 3, 2022 at 2:32 PM Krisztián Szűcs wrote: > I managed to upload the wheels to pypi. Here is the current status > with the updated assignments: > > 1. [done] make the released version

Re: Bearer Token Refresh Task

2022-02-08 Thread David Li
For gRPC, in theory, you can get UNAUTHENTICATED at any time, including after the client has gotten some results. If you need to retry calls, and you want to do it transparently, you'd need a gRPC interceptor, yes. (The Flight middleware is not powerful enough to do that.) But that should be

Bearer Token Refresh Task

2022-02-08 Thread José Almeida
Hi guys, We are assuming the Bearer Token Refresh task, which was started but now it's been paused for a while (link to original POC) <[link](https://github.com/apache/arrow/pull/8780)>, and we have some concerns about it, such as: 1 During a Flight Call we can get unauthenticated while consuming

Re: [Discuss] Best practice for storing key-value metadata for Extension Types

2022-02-08 Thread Joris Van den Bossche
On Tue, 8 Feb 2022 at 17:37, Jorge Cardoso Leitão wrote: > ... > > Wrt to binary, imo the challenge is: > * we state that backward incompatible changes to the c data interface > require a new spec [1] > Note that this discussion wouldn't change anything about the C Data Interface spec itself.

Re: [DISCUSS] New Types (Schema.fbs vs Extension Types)

2022-02-08 Thread Paul Balança
If I may, I would be really interested to be kept in the loop as well. I have been working on a small library making it easy to declare Python types and automatically getting them supported in Pyarrow as extension types (and then benefit of vecotrized ops) : https://github.com/balancap/arrowbic

Re: [DISCUSS] New Types (Schema.fbs vs Extension Types)

2022-02-08 Thread Micah Kornfield
> > I do not know if we voted on a naming convention, but we may want to > reserve a namespace for us (e.g. "arrow"). +1 to calling out in docs that the arrow namespace should be reserved. maybe "apache.arrow" to lower the possibility of collisions with people who already have extension types? (I

Re: [Discuss] Best practice for storing key-value metadata for Extension Types

2022-02-08 Thread Micah Kornfield
> > One possible alternative could be to use the format as specified in the C > Data Interface for key-value metadata: > > https://arrow.apache.org/docs/format/CDataInterface.html#c.ArrowSchema.metadata > (there it is used for the actual key-value metadata of a field, while here > it is for

Re: [Discuss] Best practice for storing key-value metadata for Extension Types

2022-02-08 Thread Jorge Cardoso Leitão
Hi, Great questions and write up. Thanks! imo dragging a JSON reader and writer to read official extension types' metadata seems overkill. The c data interface is expected to be quite low level. Imo we should aim for a (non-human readable) binary format. For non-official, imo you are spot on -

Re: [DISCUSS] New Types (Schema.fbs vs Extension Types)

2022-02-08 Thread Jorge Cardoso Leitão
Note that we do not have tests on tensor arrays, so testing the extension type on these may be hindered by divergences between implementations. I do not think we even have json integration files for them. If the focus is extension types, maybe it would be best to cover types whose physical

[Discuss] Best practice for storing key-value metadata for Extension Types

2022-02-08 Thread Joris Van den Bossche
Hi all, There is currently some discussion regarding how we can formalize/document "well known" extension types (see the "[DISCUSS] New Types (Schema.fbs vs Extension Types)" thread). There is ongoing work on an extension type to store arrays / tensors by Rok (

Re: [DISCUSS] New Types (Schema.fbs vs Extension Types)

2022-02-08 Thread Joris Van den Bossche
On Mon, 7 Feb 2022 at 21:02, Rok Mihevc wrote: > To follow up the discussion from the bi-weekly Arrow sync: > > - JSON seems the most suitable candidate for the extension metadata. > E.g.: TensorArray > {"key": "ARROW:extension:name", "value": "tensor 3, 4), strides=(12, 4, 1)>"}, > {"key":