[DISC] Improve Arrow Release verification process
Hi, One of the challenges we have when doing a release is verification and voting. Currently the Arrow verification process is quite long, tedious and error prone. I would like to use this email to get feedback and user requests in order to improve the process. Several things already on my mind: One thing that is quite annoying is that any flaky failure makes us restart the process and possibly requires downloading everything again. It would be great to have some kind of retry mechanism that allows us to keep going from where it failed and doesn't have to redo the previous successful jobs. We do have a bunch of flags to do specific parts but that requires knowledge and time to go over the different flags, etcetera so the UX could be improved. Based on the ASF release policy [1] in order to cast a +1 vote we have to validate the source code packages but it is not required to validate binaries locally. Several binaries are currently tested using docker images and they are already tested and validated on CI. Our documentation for release verification points to perform binary validation. I plan to update the documentation and move it to the official docs instead of the wiki [2]. I would appreciate input on the topic so we can improve the current process. Thanks everyone, Raúl [1] https://www.apache.org/legal/release-policy.html#release-approval [2] https://cwiki.apache.org/confluence/display/ARROW/How+to+Verify+Release+Candidates
Re: [VOTE] Release Apache Arrow 15.0.0 - RC1
+1 (binding)
Verified sources on Debian 12 'bookworm'. I had issues with binaries but that
was because of AlmaLinux failing to verify its own GPG key for some reason.
On Thu, Jan 18, 2024, at 04:40, Raúl Cumplido wrote:
> El mié, 17 ene 2024 a las 23:37, Matt Topol ()
> escribió:
>>
>> Yea, I confirmed you're correct. I have a fix for this now and I've put up
>> a PR[1] for it that Raul can cherry-pick into a new RC.
>
> As the issue seems to have been around for 1.5 or 2 years I don't
> think this issue requires a new RC. It can be cherry-picked if we want
> to do a patch release in the future.
>
>>
>> --Matt
>>
>> [1]: https://github.com/apache/arrow/pull/39674
>>
>> On Wed, Jan 17, 2024 at 2:15 PM Ruoxi Sun wrote:
>>
>> > My guts feeling is this has something to do with timezone, I'm in UTC+8.
>> >
>> > If I change this line [1] to:
>> > testTime := time.Now().UTC()
>> > test passed.
>> >
>> > So for anyone who can't reproduce, maybe you can try change [1] to:
>> > loc, _ := time.LoadLocation("Asia/Shanghai")
>> > testTime := time.Now().In(loc)
>> > to increase the chance?
>> >
>> > [1]
>> >
>> > https://github.com/apache/arrow/blob/c170af41ba0c30b80aa4172da0b3637206368cf2/go/arrow/flight/flightsql/driver/utils_test.go#L90
>> >
>> > *Regards,*
>> > *Rossi*
>> >
>> >
>> > Matt Topol 于2024年1月18日周四 02:55写道：
>> >
>> > > @pitrou Looks like the verification issue is the script just taking the
>> > > system Go if it finds it rather than verifying it's at least Go 1.19+ so
>> > it
>> > > should be pretty easy to fix that and not a release blocker.
>> > >
>> > > Regarding the unit test failure, it looks like I can't replicate it on my
>> > > linux machine and I don't have access to a mac at the moment. Would
>> > anyone
>> > > happen to have a mac they can try to dig into and check out that unit
>> > test
>> > > on? Otherwise I can spin up an AWS instance and try replicating and
>> > > debugging on that if necessary.
>> > >
>> > > --Matt
>> > >
>> > > On Wed, Jan 17, 2024 at 1:30 PM Matt Topol
>> > wrote:
>> > >
>> > > > I'll take a look at that Go test failure in a bit.
>> > > >
>> > > > As for the ubuntu 22.04 verification failure, I'll double check that
>> > > we're
>> > > > installing Go 1.19 for the verification and using the right PATH to
>> > it, I
>> > > > thought we addressed this but I guess something must have been
>> > > overlooked.
>> > > >
>> > > > --Matt
>> > > >
>> > > > On Wed, Jan 17, 2024 at 1:26 PM Ruoxi Sun
>> > wrote:
>> > > >
>> > > >> Thanks Raúl, and sorry about the confusion. I should've used RC1.
>> > > >>
>> > > >> Now I just verified RC1 but the failure remains. What's more I tried
>> > on
>> > > my
>> > > >> two laptops (one Intel Mac and one M1 Mac), and got the same failure.
>> > > >>
>> > > >>
>> > > >> *Regards,*
>> > > >> *Rossi*
>> > > >>
>> > > >>
>> > > >> Raúl Cumplido 于2024年1月18日周四 02:14写道：
>> > > >>
>> > > >> > Not sure what the issue is with the test but could you use RC1
>> > instead
>> > > >> of
>> > > >> > RC0:
>> > > >> > TEST_DEFAULT=0 TEST_GO=1 TEST_CPP=1 TEST_PYTHON=1
>> > > >> > ./verify-release-candidate.sh 15.0.0 1
>> > > >> >
>> > > >> > instead of:
>> > > >> > TEST_DEFAULT=0 TEST_GO=1 TEST_CPP=1 TEST_PYTHON=1
>> > > >> > ./verify-release-candidate.sh 15.0.0 0
>> > > >> >
>> > > >> > El mié, 17 ene 2024 a las 19:07, Ruoxi Sun (> > >)
>> > > >> > escribió:
>> > > >> > >
>> > > >> > > Tried:
>> > > >> > >
>> > > >> > > TEST_DEFAULT=0 TEST_GO=1 TEST_CPP=1 TEST_PYTHON=1
>> > > >> > > ./verify-release-candidate.sh 15.0.0 0
>> > > >> > >
>> > > >> > > But one of go test failed:
>> > > >> > >
>> > > >> > > ok github.com/apache/arrow/go/v15/arrow/flight/flightsql
>> > > >> 4.990s
>> > > >> > > 1970-01-01 12:00:00 + UTC
>> > > >> > > --- FAIL: Test_fromArrowType (0.00s)
>> > > >> > > --- FAIL: Test_fromArrowType/fromArrowType_date64_f16-d64
>> > > (0.00s)
>> > > >> > > utils_test.go:104: test failed, wanted time.Time
>> > 2024-01-17
>> > > >> > > 00:00:00 + UTC got time.Time 2024-01-18 00:00:00 + UTC
>> > > >> > > FAIL
>> > > >> > > FAIL
>> > > github.com/apache/arrow/go/v15/arrow/flight/flightsql/driver
>> > > >> > > 5.821s
>> > > >> > >
>> > > >> > > *Regards,*
>> > > >> > > *Rossi*
>> > > >> > >
>> > > >> > >
>> > > >> > > Raúl Cumplido 于2024年1月18日周四 01:29写道：
>> > > >> > >
>> > > >> > > > +1 (binding)
>> > > >> > > >
>> > > >> > > > I've verified successfully the sources and binaries with:
>> > > >> > > >
>> > > >> > > > TEST_DEFAULT=0 TEST_SOURCE=1
>> > > dev/release/verify-release-candidate.sh
>> > > >> > > > 15.0.0 1
>> > > >> > > > TEST_DEFAULT=0 TEST_BINARIES=1
>> > > >> dev/release/verify-release-candidate.sh
>> > > >> > > > 15.0.0 1
>> > > >> > > > with:
>> > > >> > > > * Python 3.10.12
>> > > >> > > > * gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
>> > > >> > > > * NVIDIA CUDA Build cuda_11.5.r11.5/compiler.30672275_0
>> > > >> > > > * openjdk version "17.0.9" 2023-10-17
>> > > >> > > > * ruby 3.0.2p107 (
Re: [VOTE] Release Apache Arrow 15.0.0 - RC1
> As the issue seems to have been around for 1.5 or 2 years I don't
think this issue requires a new RC
Fair enough.
Everything else seems good to me, +1 (binding). Verified on Ubuntu 22.04
On Fri, Jan 19, 2024 at 9:55 AM David Li wrote:
> +1 (binding)
>
> Verified sources on Debian 12 'bookworm'. I had issues with binaries but
> that was because of AlmaLinux failing to verify its own GPG key for some
> reason.
>
> On Thu, Jan 18, 2024, at 04:40, Raúl Cumplido wrote:
> > El mié, 17 ene 2024 a las 23:37, Matt Topol ()
> escribió:
> >>
> >> Yea, I confirmed you're correct. I have a fix for this now and I've put
> up
> >> a PR[1] for it that Raul can cherry-pick into a new RC.
> >
> > As the issue seems to have been around for 1.5 or 2 years I don't
> > think this issue requires a new RC. It can be cherry-picked if we want
> > to do a patch release in the future.
> >
> >>
> >> --Matt
> >>
> >> [1]: https://github.com/apache/arrow/pull/39674
> >>
> >> On Wed, Jan 17, 2024 at 2:15 PM Ruoxi Sun
> wrote:
> >>
> >> > My guts feeling is this has something to do with timezone, I'm in
> UTC+8.
> >> >
> >> > If I change this line [1] to:
> >> > testTime := time.Now().UTC()
> >> > test passed.
> >> >
> >> > So for anyone who can't reproduce, maybe you can try change [1] to:
> >> > loc, _ := time.LoadLocation("Asia/Shanghai")
> >> > testTime := time.Now().In(loc)
> >> > to increase the chance?
> >> >
> >> > [1]
> >> >
> >> >
> https://github.com/apache/arrow/blob/c170af41ba0c30b80aa4172da0b3637206368cf2/go/arrow/flight/flightsql/driver/utils_test.go#L90
> >> >
> >> > *Regards,*
> >> > *Rossi*
> >> >
> >> >
> >> > Matt Topol 于2024年1月18日周四 02:55写道：
> >> >
> >> > > @pitrou Looks like the verification issue is the script just taking
> the
> >> > > system Go if it finds it rather than verifying it's at least Go
> 1.19+ so
> >> > it
> >> > > should be pretty easy to fix that and not a release blocker.
> >> > >
> >> > > Regarding the unit test failure, it looks like I can't replicate it
> on my
> >> > > linux machine and I don't have access to a mac at the moment. Would
> >> > anyone
> >> > > happen to have a mac they can try to dig into and check out that
> unit
> >> > test
> >> > > on? Otherwise I can spin up an AWS instance and try replicating and
> >> > > debugging on that if necessary.
> >> > >
> >> > > --Matt
> >> > >
> >> > > On Wed, Jan 17, 2024 at 1:30 PM Matt Topol
> >> > wrote:
> >> > >
> >> > > > I'll take a look at that Go test failure in a bit.
> >> > > >
> >> > > > As for the ubuntu 22.04 verification failure, I'll double check
> that
> >> > > we're
> >> > > > installing Go 1.19 for the verification and using the right PATH
> to
> >> > it, I
> >> > > > thought we addressed this but I guess something must have been
> >> > > overlooked.
> >> > > >
> >> > > > --Matt
> >> > > >
> >> > > > On Wed, Jan 17, 2024 at 1:26 PM Ruoxi Sun
> >> > wrote:
> >> > > >
> >> > > >> Thanks Raúl, and sorry about the confusion. I should've used RC1.
> >> > > >>
> >> > > >> Now I just verified RC1 but the failure remains. What's more I
> tried
> >> > on
> >> > > my
> >> > > >> two laptops (one Intel Mac and one M1 Mac), and got the same
> failure.
> >> > > >>
> >> > > >>
> >> > > >> *Regards,*
> >> > > >> *Rossi*
> >> > > >>
> >> > > >>
> >> > > >> Raúl Cumplido 于2024年1月18日周四 02:14写道：
> >> > > >>
> >> > > >> > Not sure what the issue is with the test but could you use RC1
> >> > instead
> >> > > >> of
> >> > > >> > RC0:
> >> > > >> > TEST_DEFAULT=0 TEST_GO=1 TEST_CPP=1 TEST_PYTHON=1
> >> > > >> > ./verify-release-candidate.sh 15.0.0 1
> >> > > >> >
> >> > > >> > instead of:
> >> > > >> > TEST_DEFAULT=0 TEST_GO=1 TEST_CPP=1 TEST_PYTHON=1
> >> > > >> > ./verify-release-candidate.sh 15.0.0 0
> >> > > >> >
> >> > > >> > El mié, 17 ene 2024 a las 19:07, Ruoxi Sun (<
> zanmato1...@gmail.com
> >> > >)
> >> > > >> > escribió:
> >> > > >> > >
> >> > > >> > > Tried:
> >> > > >> > >
> >> > > >> > > TEST_DEFAULT=0 TEST_GO=1 TEST_CPP=1 TEST_PYTHON=1
> >> > > >> > > ./verify-release-candidate.sh 15.0.0 0
> >> > > >> > >
> >> > > >> > > But one of go test failed:
> >> > > >> > >
> >> > > >> > > ok
> github.com/apache/arrow/go/v15/arrow/flight/flightsql
> >> > > >> 4.990s
> >> > > >> > > 1970-01-01 12:00:00 + UTC
> >> > > >> > > --- FAIL: Test_fromArrowType (0.00s)
> >> > > >> > > --- FAIL: Test_fromArrowType/fromArrowType_date64_f16-d64
> >> > > (0.00s)
> >> > > >> > > utils_test.go:104: test failed, wanted time.Time
> >> > 2024-01-17
> >> > > >> > > 00:00:00 + UTC got time.Time 2024-01-18 00:00:00 +
> UTC
> >> > > >> > > FAIL
> >> > > >> > > FAIL
> >> > > github.com/apache/arrow/go/v15/arrow/flight/flightsql/driver
> >> > > >> > > 5.821s
> >> > > >> > >
> >> > > >> > > *Regards,*
> >> > > >> > > *Rossi*
> >> > > >> > >
> >> > > >> > >
> >> > > >> > > Raúl Cumplido 于2024年1月18日周四 01:29写道：
> >> > > >> > >
> >> > > >> > > > +1 (binding)
> >> > > >> > > >
> >> > > >> > > > I've verified successfully the sources a
Re: [DISCUSS][MATLAB] Proposed "Category B" License for Bundling MATLAB MEX Build Artifacts in Official Arrow Release
On Thu, Jan 18, 2024 at 12:24 PM Ian Cook wrote: > > Hi Sarah, > > Thanks for pursuing this. > > The ASF 3rd Party License Policy lists a number of standard, > off-the-shelf licenses that are compatible with Category B, but the > policy does not include any provision for custom-written licenses. > This appears to be a custom-written license. Is that correct? > > Is this custom-written license based on one of the listed Category B > licenses? If so, can you tell us which one? If not, can you provide > some explanation of why this license should be considered to meet the > criteria for Category B? FWIW: while these are all excellent questions for the pre-work, if there needs to be an ultimate statement on this -- you'll have to file a LEGAL JIRA. E.g.: https://issues.apache.org/jira/browse/LEGAL-506 (plz include all the relevant details when filing it -- whatever comes out of this thread). Thanks, Roman. > > Thank you, > Ian > > On Wed, Jan 17, 2024 at 12:08 PM Sarah Gilmore > wrote: > > > > Hi Everyone, > > > > Kevin Gurney and I have been working on integrating the MATLAB Arrow > > bindings with the project's release processes in this pull request [1]. > > While working on integrating with the release tooling, we realized that we > > need to ensure that the licenses of any MEX artifacts [2] bundled with the > > released MLTBX [3] file are compatible with the ASF 3rd Party License > > Policy [4]. > > > > After several rounds of discussion with some colleagues at MathWorks, we > > came up with a license [5] that is intended to meet the requirements for > > inclusion as a "Category B" [6] license according to the ASF 3rd Party > > License Policy. > > > > Our goal is to make sure we are doing the right thing here, so, as per > > Kou's suggestion [7], we wanted to share the proposed license [5] with the > > broader Arrow development community. We understand this may need further > > input from ASF Legal as well. > > > > Please let us know what we can do to help move this forward. We sincerely > > appreciate everyone's support as we navigate these licensing requirements. > > > > [1] https://github.com/apache/arrow/pull/38660 > > [2] https://www.mathworks.com/help/matlab/call-mex-functions.html > > [3] > > https://www.mathworks.com/help/matlab/creating-help.html?s_tid=CRUX_lftnav > > [4] https://www.apache.org/legal/resolved.html > > [5] https://github.com/apache/arrow/files/13955180/license.txt > > [6] https://www.apache.org/legal/resolved.html#category-b > > [7] https://github.com/apache/arrow/pull/38660#discussion_r1454804607 > > > > Best, > > > > Sarah Gilmore > >
Re: [VOTE] Release Apache Arrow 15.0.0 - RC1
El vie, 19 ene 2024 a las 15:56, David Li () escribió:
>
> +1 (binding)
>
> Verified sources on Debian 12 'bookworm'. I had issues with binaries but that
> was because of AlmaLinux failing to verify its own GPG key for some reason.
I had the same issue locally. I fixed it by removing the local docker
image I had for almalinux:8 and pulling again.
>
> On Thu, Jan 18, 2024, at 04:40, Raúl Cumplido wrote:
> > El mié, 17 ene 2024 a las 23:37, Matt Topol ()
> > escribió:
> >>
> >> Yea, I confirmed you're correct. I have a fix for this now and I've put up
> >> a PR[1] for it that Raul can cherry-pick into a new RC.
> >
> > As the issue seems to have been around for 1.5 or 2 years I don't
> > think this issue requires a new RC. It can be cherry-picked if we want
> > to do a patch release in the future.
> >
> >>
> >> --Matt
> >>
> >> [1]: https://github.com/apache/arrow/pull/39674
> >>
> >> On Wed, Jan 17, 2024 at 2:15 PM Ruoxi Sun wrote:
> >>
> >> > My guts feeling is this has something to do with timezone, I'm in UTC+8.
> >> >
> >> > If I change this line [1] to:
> >> > testTime := time.Now().UTC()
> >> > test passed.
> >> >
> >> > So for anyone who can't reproduce, maybe you can try change [1] to:
> >> > loc, _ := time.LoadLocation("Asia/Shanghai")
> >> > testTime := time.Now().In(loc)
> >> > to increase the chance?
> >> >
> >> > [1]
> >> >
> >> > https://github.com/apache/arrow/blob/c170af41ba0c30b80aa4172da0b3637206368cf2/go/arrow/flight/flightsql/driver/utils_test.go#L90
> >> >
> >> > *Regards,*
> >> > *Rossi*
> >> >
> >> >
> >> > Matt Topol 于2024年1月18日周四 02:55写道：
> >> >
> >> > > @pitrou Looks like the verification issue is the script just taking the
> >> > > system Go if it finds it rather than verifying it's at least Go 1.19+
> >> > > so
> >> > it
> >> > > should be pretty easy to fix that and not a release blocker.
> >> > >
> >> > > Regarding the unit test failure, it looks like I can't replicate it on
> >> > > my
> >> > > linux machine and I don't have access to a mac at the moment. Would
> >> > anyone
> >> > > happen to have a mac they can try to dig into and check out that unit
> >> > test
> >> > > on? Otherwise I can spin up an AWS instance and try replicating and
> >> > > debugging on that if necessary.
> >> > >
> >> > > --Matt
> >> > >
> >> > > On Wed, Jan 17, 2024 at 1:30 PM Matt Topol
> >> > wrote:
> >> > >
> >> > > > I'll take a look at that Go test failure in a bit.
> >> > > >
> >> > > > As for the ubuntu 22.04 verification failure, I'll double check that
> >> > > we're
> >> > > > installing Go 1.19 for the verification and using the right PATH to
> >> > it, I
> >> > > > thought we addressed this but I guess something must have been
> >> > > overlooked.
> >> > > >
> >> > > > --Matt
> >> > > >
> >> > > > On Wed, Jan 17, 2024 at 1:26 PM Ruoxi Sun
> >> > wrote:
> >> > > >
> >> > > >> Thanks Raúl, and sorry about the confusion. I should've used RC1.
> >> > > >>
> >> > > >> Now I just verified RC1 but the failure remains. What's more I tried
> >> > on
> >> > > my
> >> > > >> two laptops (one Intel Mac and one M1 Mac), and got the same
> >> > > >> failure.
> >> > > >>
> >> > > >>
> >> > > >> *Regards,*
> >> > > >> *Rossi*
> >> > > >>
> >> > > >>
> >> > > >> Raúl Cumplido 于2024年1月18日周四 02:14写道：
> >> > > >>
> >> > > >> > Not sure what the issue is with the test but could you use RC1
> >> > instead
> >> > > >> of
> >> > > >> > RC0:
> >> > > >> > TEST_DEFAULT=0 TEST_GO=1 TEST_CPP=1 TEST_PYTHON=1
> >> > > >> > ./verify-release-candidate.sh 15.0.0 1
> >> > > >> >
> >> > > >> > instead of:
> >> > > >> > TEST_DEFAULT=0 TEST_GO=1 TEST_CPP=1 TEST_PYTHON=1
> >> > > >> > ./verify-release-candidate.sh 15.0.0 0
> >> > > >> >
> >> > > >> > El mié, 17 ene 2024 a las 19:07, Ruoxi Sun ( >> > >)
> >> > > >> > escribió:
> >> > > >> > >
> >> > > >> > > Tried:
> >> > > >> > >
> >> > > >> > > TEST_DEFAULT=0 TEST_GO=1 TEST_CPP=1 TEST_PYTHON=1
> >> > > >> > > ./verify-release-candidate.sh 15.0.0 0
> >> > > >> > >
> >> > > >> > > But one of go test failed:
> >> > > >> > >
> >> > > >> > > ok github.com/apache/arrow/go/v15/arrow/flight/flightsql
> >> > > >> 4.990s
> >> > > >> > > 1970-01-01 12:00:00 + UTC
> >> > > >> > > --- FAIL: Test_fromArrowType (0.00s)
> >> > > >> > > --- FAIL: Test_fromArrowType/fromArrowType_date64_f16-d64
> >> > > (0.00s)
> >> > > >> > > utils_test.go:104: test failed, wanted time.Time
> >> > 2024-01-17
> >> > > >> > > 00:00:00 + UTC got time.Time 2024-01-18 00:00:00 + UTC
> >> > > >> > > FAIL
> >> > > >> > > FAIL
> >> > > github.com/apache/arrow/go/v15/arrow/flight/flightsql/driver
> >> > > >> > > 5.821s
> >> > > >> > >
> >> > > >> > > *Regards,*
> >> > > >> > > *Rossi*
> >> > > >> > >
> >> > > >> > >
> >> > > >> > > Raúl Cumplido 于2024年1月18日周四 01:29写道：
> >> > > >> > >
> >> > > >> > > > +1 (binding)
> >> > > >> > > >
> >> > > >> > > > I've verified successfully the sources and binaries with:
> >> > > >> > > >
> >> > > >> > > > TEST_DEFAULT=0 TEST_SOURCE=
Re: [DISC] Improve Arrow Release verification process
I agree that this is a great time to look at improving the verification process. One solution I've seen work fairly well is to convert large bash scripts into a lightweight ETL pipeline that caches the results/status of each node as it executes. That way, restarting a pipeline at the right checkpoint is trivial. Existing open source ETL platforms should be able to do this, although I don't know which would be best. Something like Apache Airflow would probably be overkill IMO. On Fri, Jan 19, 2024 at 5:44 AM Raúl Cumplido wrote: > Hi, > > One of the challenges we have when doing a release is verification and > voting. > > Currently the Arrow verification process is quite long, tedious and error > prone. > > I would like to use this email to get feedback and user requests in > order to improve the process. > > Several things already on my mind: > > One thing that is quite annoying is that any flaky failure makes us > restart the process and possibly requires downloading everything > again. It would be great to have some kind of retry mechanism that > allows us to keep going from where it failed and doesn't have to redo > the previous successful jobs. > > We do have a bunch of flags to do specific parts but that requires > knowledge and time to go over the different flags, etcetera so the UX > could be improved. > > Based on the ASF release policy [1] in order to cast a +1 vote we have > to validate the source code packages but it is not required to > validate binaries locally. Several binaries are currently tested using > docker images and they are already tested and validated on CI. Our > documentation for release verification points to perform binary > validation. I plan to update the documentation and move it to the > official docs instead of the wiki [2]. > > I would appreciate input on the topic so we can improve the current > process. > > Thanks everyone, > Raúl > > [1] https://www.apache.org/legal/release-policy.html#release-approval > [2] > https://cwiki.apache.org/confluence/display/ARROW/How+to+Verify+Release+Candidates >
Re: [DISC] Improve Arrow Release verification process
Well, if the main objective is to just follow the ASF Release guidelines, then our verification process can be simplified drastically. The ASF indeed just requires: """ Every ASF release MUST contain one or more source packages, which MUST be sufficient for a user to build and test the release provided they have access to the appropriate platform and tools. A source release SHOULD not contain compiled code. """ So, basically, if the source tarball is enough to compile Arrow on a single platform with a single set of tools, then we're ok. :-) The rest is just an additional burden that we voluntarily inflict to ourselves. *Ideally*, our continuous integration should be able to detect any build-time or runtime issue on supported platforms. There have been rare cases where some issues could be detected at release time thanks to the release verification script, but these are a tiny portion of all issues routinely detected in the form of CI failures. So, there doesn't seem to be a reason to believe that manual release verification is bringing significant benefits compared to regular CI. Regards Antoine. Le 19/01/2024 à 11:42, Raúl Cumplido a écrit : Hi, One of the challenges we have when doing a release is verification and voting. Currently the Arrow verification process is quite long, tedious and error prone. I would like to use this email to get feedback and user requests in order to improve the process. Several things already on my mind: One thing that is quite annoying is that any flaky failure makes us restart the process and possibly requires downloading everything again. It would be great to have some kind of retry mechanism that allows us to keep going from where it failed and doesn't have to redo the previous successful jobs. We do have a bunch of flags to do specific parts but that requires knowledge and time to go over the different flags, etcetera so the UX could be improved. Based on the ASF release policy [1] in order to cast a +1 vote we have to validate the source code packages but it is not required to validate binaries locally. Several binaries are currently tested using docker images and they are already tested and validated on CI. Our documentation for release verification points to perform binary validation. I plan to update the documentation and move it to the official docs instead of the wiki [2]. I would appreciate input on the topic so we can improve the current process. Thanks everyone, Raúl [1] https://www.apache.org/legal/release-policy.html#release-approval [2] https://cwiki.apache.org/confluence/display/ARROW/How+to+Verify+Release+Candidates
Re: [DISC] Improve Arrow Release verification process
I would second this notion that manually running tests that are already covered as part of CI as part of the release process is of (very) limited value. While we do the same thing (compile and run some tests) as part of the Rust release this has never caught any serious defect I am aware of and we only run a subset of tests (e.g. not tests for integration with other arrow versions) Reducing the burden for releases I think would benefit everyone. Andrew On Fri, Jan 19, 2024 at 1:08 PM Antoine Pitrou wrote: > > Well, if the main objective is to just follow the ASF Release > guidelines, then our verification process can be simplified drastically. > > The ASF indeed just requires: > """ > Every ASF release MUST contain one or more source packages, which MUST > be sufficient for a user to build and test the release provided they > have access to the appropriate platform and tools. A source release > SHOULD not contain compiled code. > """ > > So, basically, if the source tarball is enough to compile Arrow on a > single platform with a single set of tools, then we're ok. :-) > > The rest is just an additional burden that we voluntarily inflict to > ourselves. *Ideally*, our continuous integration should be able to > detect any build-time or runtime issue on supported platforms. There > have been rare cases where some issues could be detected at release time > thanks to the release verification script, but these are a tiny portion > of all issues routinely detected in the form of CI failures. So, there > doesn't seem to be a reason to believe that manual release verification > is bringing significant benefits compared to regular CI. > > Regards > > Antoine. > > > Le 19/01/2024 à 11:42, Raúl Cumplido a écrit : > > Hi, > > > > One of the challenges we have when doing a release is verification and > voting. > > > > Currently the Arrow verification process is quite long, tedious and > error prone. > > > > I would like to use this email to get feedback and user requests in > > order to improve the process. > > > > Several things already on my mind: > > > > One thing that is quite annoying is that any flaky failure makes us > > restart the process and possibly requires downloading everything > > again. It would be great to have some kind of retry mechanism that > > allows us to keep going from where it failed and doesn't have to redo > > the previous successful jobs. > > > > We do have a bunch of flags to do specific parts but that requires > > knowledge and time to go over the different flags, etcetera so the UX > > could be improved. > > > > Based on the ASF release policy [1] in order to cast a +1 vote we have > > to validate the source code packages but it is not required to > > validate binaries locally. Several binaries are currently tested using > > docker images and they are already tested and validated on CI. Our > > documentation for release verification points to perform binary > > validation. I plan to update the documentation and move it to the > > official docs instead of the wiki [2]. > > > > I would appreciate input on the topic so we can improve the current > process. > > > > Thanks everyone, > > Raúl > > > > [1] https://www.apache.org/legal/release-policy.html#release-approval > > [2] > https://cwiki.apache.org/confluence/display/ARROW/How+to+Verify+Release+Candidates >
Re: [DISCUSS][MATLAB] Proposed "Category B" License for Bundling MATLAB MEX Build Artifacts in Official Arrow Release
Hi Roman, > FWIW: while these are all excellent questions for the pre-work, if there > needs to be an ultimate statement on this -- you'll have to file a LEGAL > JIRA. E.g.: https://issues.apache.org/jira/browse/LEGAL-506 > > (plz include all the relevant details when filing it -- whatever comes > out of this thread). Thank you for the guidance. We suspected this may be the case and will be sure to include all the relevant information when we file the Jira issue. Best, Sarah and Kevin From: Roman Shaposhnik Sent: Friday, January 19, 2024 12:15 PM To: dev@arrow.apache.org Subject: Re: [DISCUSS][MATLAB] Proposed "Category B" License for Bundling MATLAB MEX Build Artifacts in Official Arrow Release On Thu, Jan 18, 2024 at 12:24 PM Ian Cook wrote: > > Hi Sarah, > > Thanks for pursuing this. > > The ASF 3rd Party License Policy lists a number of standard, > off-the-shelf licenses that are compatible with Category B, but the > policy does not include any provision for custom-written licenses. > This appears to be a custom-written license. Is that correct? > > Is this custom-written license based on one of the listed Category B > licenses? If so, can you tell us which one? If not, can you provide > some explanation of why this license should be considered to meet the > criteria for Category B? FWIW: while these are all excellent questions for the pre-work, if there needs to be an ultimate statement on this -- you'll have to file a LEGAL JIRA. E.g.: https://issues.apache.org/jira/browse/LEGAL-506 (plz include all the relevant details when filing it -- whatever comes out of this thread). Thanks, Roman. > > Thank you, > Ian > > On Wed, Jan 17, 2024 at 12:08 PM Sarah Gilmore > wrote: > > > > Hi Everyone, > > > > Kevin Gurney and I have been working on integrating the MATLAB Arrow > > bindings with the project's release processes in this pull request [1]. > > While working on integrating with the release tooling, we realized that we > > need to ensure that the licenses of any MEX artifacts [2] bundled with the > > released MLTBX [3] file are compatible with the ASF 3rd Party License > > Policy [4]. > > > > After several rounds of discussion with some colleagues at MathWorks, we > > came up with a license [5] that is intended to meet the requirements for > > inclusion as a "Category B" [6] license according to the ASF 3rd Party > > License Policy. > > > > Our goal is to make sure we are doing the right thing here, so, as per > > Kou's suggestion [7], we wanted to share the proposed license [5] with the > > broader Arrow development community. We understand this may need further > > input from ASF Legal as well. > > > > Please let us know what we can do to help move this forward. We sincerely > > appreciate everyone's support as we navigate these licensing requirements. > > > > [1] https://github.com/apache/arrow/pull/38660 > > [2] https://www.mathworks.com/help/matlab/call-mex-functions.html > > [3] > > https://www.mathworks.com/help/matlab/creating-help.html?s_tid=CRUX_lftnav > > [4] https://www.apache.org/legal/resolved.html > > [5] https://github.com/apache/arrow/files/13955180/license.txt > > [6] https://www.apache.org/legal/resolved.html#category-b > > [7] https://github.com/apache/arrow/pull/38660#discussion_r1454804607 > > > > Best, > > > > Sarah Gilmore > >
Re: [DISC] Improve Arrow Release verification process
I concur, a minimally scoped verification script for the actual voting process without any binary verification etc. should be created. The ease in verifying a release will lower the burden to participate in the vote which is good for the community and will even be necessary if we ever want to increase release cadance as previously discussed. In my opinion it will also mean that the binaries are no longer part of the release, which will help in situations similar to the release of Python 3.12 just after 14.0.0 was released and lots of users were running into issues because there were no 14.0.0 wheels for 3.12. While it would still be nice to potentially make reproduction of CI errors easier by having better methods to restart a failed script, this is of much lower importance then improving the release process. Jacob On Fri, Jan 19, 2024 at 7:38 PM Andrew Lamb wrote: > I would second this notion that manually running tests that are already > covered as part of CI as part of the release process is of (very) limited > value. > > While we do the same thing (compile and run some tests) as part of the Rust > release this has never caught any serious defect I am aware of and we only > run a subset of tests (e.g. not tests for integration with other arrow > versions) > > Reducing the burden for releases I think would benefit everyone. > > Andrew > > On Fri, Jan 19, 2024 at 1:08 PM Antoine Pitrou wrote: > > > > > Well, if the main objective is to just follow the ASF Release > > guidelines, then our verification process can be simplified drastically. > > > > The ASF indeed just requires: > > """ > > Every ASF release MUST contain one or more source packages, which MUST > > be sufficient for a user to build and test the release provided they > > have access to the appropriate platform and tools. A source release > > SHOULD not contain compiled code. > > """ > > > > So, basically, if the source tarball is enough to compile Arrow on a > > single platform with a single set of tools, then we're ok. :-) > > > > The rest is just an additional burden that we voluntarily inflict to > > ourselves. *Ideally*, our continuous integration should be able to > > detect any build-time or runtime issue on supported platforms. There > > have been rare cases where some issues could be detected at release time > > thanks to the release verification script, but these are a tiny portion > > of all issues routinely detected in the form of CI failures. So, there > > doesn't seem to be a reason to believe that manual release verification > > is bringing significant benefits compared to regular CI. > > > > Regards > > > > Antoine. > > > > > > Le 19/01/2024 à 11:42, Raúl Cumplido a écrit : > > > Hi, > > > > > > One of the challenges we have when doing a release is verification and > > voting. > > > > > > Currently the Arrow verification process is quite long, tedious and > > error prone. > > > > > > I would like to use this email to get feedback and user requests in > > > order to improve the process. > > > > > > Several things already on my mind: > > > > > > One thing that is quite annoying is that any flaky failure makes us > > > restart the process and possibly requires downloading everything > > > again. It would be great to have some kind of retry mechanism that > > > allows us to keep going from where it failed and doesn't have to redo > > > the previous successful jobs. > > > > > > We do have a bunch of flags to do specific parts but that requires > > > knowledge and time to go over the different flags, etcetera so the UX > > > could be improved. > > > > > > Based on the ASF release policy [1] in order to cast a +1 vote we have > > > to validate the source code packages but it is not required to > > > validate binaries locally. Several binaries are currently tested using > > > docker images and they are already tested and validated on CI. Our > > > documentation for release verification points to perform binary > > > validation. I plan to update the documentation and move it to the > > > official docs instead of the wiki [2]. > > > > > > I would appreciate input on the topic so we can improve the current > > process. > > > > > > Thanks everyone, > > > Raúl > > > > > > [1] https://www.apache.org/legal/release-policy.html#release-approval > > > [2] > > > https://cwiki.apache.org/confluence/display/ARROW/How+to+Verify+Release+Candidates > > >
[IPC] Delta Dictionary Flag Clarification for Multi-Batch IPC
Hi folks, I'm working on multi-batch dictionary with delta support in Java [1] and would like some clarifications. Given the "isDelta" flag in the dictionary message [2], when should this be set to "true"? 1) If we have dictionary with an ID of 1 that we want to delta encode and it is used across multiple batches, should the initial batch have `isDelta=false` then subsequent batches have `isDelta=true`? E.g. batch 1, dict 1, isDelta=false, dictVector=[a, b, c], indexVector=[0, 1, 1, 2] batch 2, dict 1, isDelta=true, dictVector=[d], indexVector=[2, 3, 0, 1] batch 3, dict 1, isDelta=true, dictVector=[e], indexVector=[0, 4] Or should the flag be true for the entire IPC flow? E.g. batch 1, dict 1, isDelta=true, dictVector=[a, b, c], indexVector=[0, 1, 1, 2] batch 2, dict 1, isDelta=true, dictVector=[d], indexVector=[2, 3, 0, 1] batch 3, dict 1, isDelta=true, dictVector=[e], indexVector=[0, 4, 3] Either works for me. 2) Could (in stream, not file IPCs) a single dictionary ever switch state across batches from delta to replacement mode or vice-versa? E.g. batch 1, dict 1, isDelta = true, dictVector=[a, b, c], indexVector=[0, 1, 1, 2] batch 2, dict 1, isDelta = true, dictVector=[d], indexVector=[2, 3, 0, 1] batch 3, dict 1, isDelta = false, dictVector=[c, a, d], indexVector=[0, 1, 2] I'd like to keep the protocol and API simple and assume switching is not allowed. This would mean the 2nd example above would be canonical. 3) Are replacement dictionaries required to be serialized for every batch or is a dictionary re-used across batches until a replacement is received? The CPP IPC API has 'unify_dictionaries' [3] that mentions "a column with a dictionary type must have the same dictionary in each record batch". I assume (and prefer) the latter, that replacements are serialized once and re-used. E.g. batch 1, dict 1, isDelta = false, dictVector=[a, b, c], indexVector=[0, 1, 1, 2] batch 2, dict 1, isDelta = false, dictVector=[], indexVector=[2, 1, 0, 1] // use previous dictionary batch 3, dict 1, isDelta = false, dictVector=[c, a, d], indexVector=[0, 1, 2] // replacement And I assume that 'unify_dictionaries' simply concatenates all dictionaries into a single vector serialized in the first batch (haven't looked at the code yet). 4) Is it valid for a delta dictionary to have an update in a subsequent batch even though the update is not used in that batch? A silly example would be: batch 1, dict 1, isDelta = true, dictVector=[a, b, c], indexVector=[0, 1, 1, 2] batch 2, dict 1, isDelta = true, dictVector=[d], indexVector=[null, null, null, null] batch 3, dict 1, isDelta = true, dictVector=[], indexVector=[0, 3, 2] Thanks for your help! [1] https://github.com/apache/arrow/pull/38423 [2] https://github.com/apache/arrow/blob/main/format/Message.fbs#L134 [3] https://arrow.apache.org/docs/cpp/api/ipc.html#_CPPv4N5arrow3ipc15IpcWriteOptions18unify_dictionariesE -- Chris Larsen
