Taking a look now. The rest of the backlog is closed out so as far as I'm concerned the RC can go out anytime
On Thu, Jan 10, 2019 at 12:33 PM Antoine Pitrou <anto...@python.org> wrote: > > > I've added https://issues.apache.org/jira/browse/ARROW-4216 ("Python > CUDA API docs") to the 0.12.0 release. There's a PR ready and it > doesn't look contentious. > > Regards > > Antoine. > > > Le 10/01/2019 à 00:40, Wes McKinney a écrit : > > I'll connect with Krisztian tomorrow (Thursday) morning to make sure > > KEYS are in order and to give him the day to do a run through of the > > release process to make sure there are no problems. In the meantime > > I'll try to resolve as many of the remaining issues as possible, then > > move the leftovers to 0.13. So we can start a vote on Friday > > > > On Wed, Jan 9, 2019 at 7:44 AM Antoine Pitrou <anto...@python.org> wrote: > >> > >> > >> No preference from me. > >> > >> Regards > >> > >> Antoine. > >> > >> > >> Le 09/01/2019 à 14:34, Krisztián Szűcs a écrit : > >>> Should We aim for Friday (2019-01-11) or Thursday (2019-01-10)? > >>> > >>> On Wed, Jan 9, 2019 at 11:43 AM Antoine Pitrou <anto...@python.org> wrote: > >>> > >>>> > >>>> For the record, none of these issues seem release-critical to me, apart > >>>> from the RAT issue and perhaps the Windows git-describe issue. Though > >>>> perhaps the release manager is best placed to evaluate them. > >>>> > >>>> Regards > >>>> > >>>> Antoine. > >>>> > >>>> > >>>> Le 08/01/2019 à 23:07, Wes McKinney a écrit : > >>>>> Well, a mid-week release candidate isn't looking too likely. I've > >>>>> spent most of today working on the Gandiva Windows build and haven't > >>>>> been able to do much 0.12 backlog items yesterday or today so far. > >>>>> > >>>>> Krisztian -- is your GPG key in KEYS yet? If not, you cannot cut a > >>>> release. > >>>>> > >>>>> Out of the remaining issues in the backlog > >>>>> https://issues.apache.org/jira/projects/ARROW/versions/12343858 > >>>>> > >>>>> Python issues > >>>>> > >>>>> * ARROW-2038: small comment to be addressed, then merged. Someone > >>>>> please create follow up JIRA about CI for S3 > >>>>> * ARROW-2298: I just nixed this for 0.12. There is a patch up but it > >>>>> needs more work > >>>>> * ARROW-2659 and ARROW-2860 are the same issue, I think. We should > >>>>> fix as it has impacted many users and been reported many times > >>>>> * ARROW-3344: Recurring Plasma test failure on Ubuntu 14.04. I will > >>>>> take a look and see if the fix is difficult or not > >>>>> * ARROW-3428: Close to merge ready, I will review and make sure all is > >>>> good > >>>>> * ARROW-4138: Windows rough edge, I will look > >>>>> * ARROW-3916: There is a patch here; can someone look? Given the > >>>>> overlap between Arrow and Parquet users it would be good to fix this > >>>>> * ARROW-4181: Failing large_memory test -- can someone look (and also > >>>>> see if any other large_memory tests are failing)? > >>>>> > >>>>> Rust items: can be merged but need not block release > >>>>> > >>>>> * ARROW-4040 > >>>>> * ARROW-4193 > >>>>> > >>>>> The other issues > >>>>> > >>>>> * ARROW-3578: The release manager will need to be careful to see what > >>>>> happens with RAT in the build > >>>>> * ARROW-4199: patch available > >>>>> * ARROW-854: To be merged with experimental designation on green build > >>>>> * ARROW-4197: Emscripten issues with C++. Would be good to merge fix > >>>>> in time for release > >>>>> > >>>>> I would say we should close the backlog by end-of-week at latest and > >>>>> move forward with the release > >>>>> > >>>>> Thanks > >>>>> Wes > >>>>> > >>>>> On Mon, Jan 7, 2019 at 10:08 AM Krisztián Szűcs > >>>>> <szucs.kriszt...@gmail.com> wrote: > >>>>>> > >>>>>> Cool. I'm going to help with the python issues right after fixing the > >>>> spark > >>>>>> integration tests. > >>>>>> > >>>>>> On Mon, Jan 7, 2019 at 4:43 PM Wes McKinney <wesmck...@gmail.com> > >>>> wrote: > >>>>>> > >>>>>>> Great, thank you! > >>>>>>> > >>>>>>> There are 20 0.12 issues Open or In-Progress, I'm going to tackle a > >>>>>>> couple more Python things today and tomorrow, but let's see where we > >>>>>>> stand by Wednesday or so and decide when to cut the release > >>>>>>> > >>>>>>> On Mon, Jan 7, 2019 at 4:38 AM Krisztián Szűcs > >>>>>>> <szucs.kriszt...@gmail.com> wrote: > >>>>>>>> > >>>>>>>> Hey! > >>>>>>>> > >>>>>>>> On Fri, Jan 4, 2019 at 10:31 PM Wes McKinney <wesmck...@gmail.com> > >>>>>>> wrote: > >>>>>>>> > >>>>>>>>> hi all, > >>>>>>>>> > >>>>>>>>> We should try to cut a release candidate for 0.12 as soon as > >>>>>>>>> practical. Since we're just coming off the holidays, it would be > >>>>>>>>> good > >>>>>>>>> to work for a few more business days to close out as many > >>>>>>>>> outstanding > >>>>>>>>> patches as possible, and be in position to start a vote sometime > >>>>>>>>> next > >>>>>>>>> week. > >>>>>>>>> > >>>>>>>>> There's a bunch of Python bugs in the backlog still -- if anyone can > >>>>>>>>> pick up one or two of these it would be a help > >>>>>>>>> > >>>>>>>>> Would someone (Krisztian or Antoine maybe?) like to be the release > >>>>>>> manager? > >>>>>>>>> > >>>>>>>> Yes, I volunteer :) > >>>>>>>> > >>>>>>>>> > >>>>>>>>> Thanks > >>>>>>>>> > >>>>>>>>> On Wed, Dec 12, 2018 at 8:15 AM Wes McKinney <wesmck...@gmail.com> > >>>>>>> wrote: > >>>>>>>>>> > >>>>>>>>>> I agree that we should aim for time-based releases. Let's discuss a > >>>>>>>>>> time-based release schedule (my preference would be ~every 2 > >>>>>>>>>> months) > >>>>>>>>>> for 2019 after we get 0.12 out. > >>>>>>>>>> On Wed, Dec 12, 2018 at 3:15 AM Antoine Pitrou <anto...@python.org> > >>>>>>>>> wrote: > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> I think we should aim for time-based releases in general (rather > >>>>>>> than a > >>>>>>>>>>> specific set of features), but delaying this one sounds good to > >>>>>>>>>>> me. > >>>>>>>>>>> > >>>>>>>>>>> Regards > >>>>>>>>>>> > >>>>>>>>>>> Antoine. > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> Le 12/12/2018 à 01:34, Wes McKinney a écrit : > >>>>>>>>>>>> hi all, > >>>>>>>>>>>> > >>>>>>>>>>>> I'm looking at the 0.12 backlog and I am not too comfortable > >>>>>>> with the > >>>>>>>>>>>> things that would have to be cut to get a release out next week. > >>>>>>>>>>>> Additionally, not a lot of developers are going to be working the > >>>>>>>>> week > >>>>>>>>>>>> of December 24 because of the Christmas and New Year's holidays, > >>>>>>> so > >>>>>>>>>>>> even if we did release, it might not get seen by a lot of people > >>>>>>>>> until > >>>>>>>>>>>> after the New Year. > >>>>>>>>>>>> > >>>>>>>>>>>> Based on this, I would suggest we push to complete as much work > >>>>>>> as > >>>>>>>>>>>> possible (from the 0.12 backlog and beyond) by the end of the > >>>>>>> year, > >>>>>>>>>>>> and release as soon as possible in 2019. Of course, anyone is > >>>>>>> welcome > >>>>>>>>>>>> to contribute work that is not found in the 0.12 milestone =) > >>>>>>>>>>>> > >>>>>>>>>>>> Any objections? > >>>>>>>>>>>> > >>>>>>>>>>>> Thanks > >>>>>>>>>>>> Wes > >>>>>>>>>>>> On Mon, Dec 10, 2018 at 8:04 AM Andy Grove < > >>>>>>> andygrov...@gmail.com> > >>>>>>>>> wrote: > >>>>>>>>>>>>> > >>>>>>>>>>>>> Cool. I will continue to add primitive operations but I am now > >>>>>>>>> adding this > >>>>>>>>>>>>> in a separate source file to keep it separate from the core > >>>>>>> array > >>>>>>>>> code. > >>>>>>>>>>>>> > >>>>>>>>>>>>> I'm not sure how important it will be to support Rust data > >>>>>>> sources > >>>>>>>>> with > >>>>>>>>>>>>> Gandiva. I can see that each language should be able to > >>>>>>> construct > >>>>>>>>> the > >>>>>>>>>>>>> logical query plan to submit to Gandiva and let Gandiva handle > >>>>>>>>> execution. I > >>>>>>>>>>>>> think the more interesting part is how do we support > >>>>>>>>> language-specific > >>>>>>>>>>>>> lambda functions as part of that logical query plan. Maybe it is > >>>>>>>>> possible > >>>>>>>>>>>>> to compile the lambda down to LLVM (I haven't started learning > >>>>>>>>> about LLVM > >>>>>>>>>>>>> in detail yet so this is wild speculation on my part). Another > >>>>>>>>> option is > >>>>>>>>>>>>> for Gandiva to support calling into shared libraries and that > >>>>>>> maybe > >>>>>>>>> is > >>>>>>>>>>>>> simpler for languages that support building C-native shared > >>>>>>>>> libraries (Rust > >>>>>>>>>>>>> supports this with zero overhead). > >>>>>>>>>>>>> > >>>>>>>>>>>>> Andy. > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> On Sun, Dec 9, 2018 at 11:42 AM Wes McKinney < > >>>>>>> wesmck...@gmail.com> > >>>>>>>>> wrote: > >>>>>>>>>>>>> > >>>>>>>>>>>>>> hi Andy, > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> I can see an argument for having some basic native function > >>>>>>> kernel > >>>>>>>>>>>>>> support in Rust. One of the things that Gandiva has begun is a > >>>>>>>>>>>>>> Protobuf-based serialized representation representation of > >>>>>>>>> projection > >>>>>>>>>>>>>> and filter expressions. In the long run I would like to see a > >>>>>>> more > >>>>>>>>>>>>>> complete relational algebra / logical query plan that can be > >>>>>>>>> submitted > >>>>>>>>>>>>>> for execution. There's complexities, though, such as bridging > >>>>>>>>>>>>>> iteration of data sources written in Rust, say, with a query > >>>>>>> engine > >>>>>>>>>>>>>> written in C++. You would need to provide some kind of a > >>>>>>> callback > >>>>>>>>>>>>>> mechanism for the query engine to request the next chunk of a > >>>>>>>>> dataset > >>>>>>>>>>>>>> to be materialized. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> It will be interested to see what contributors will be > >>>>>>> motivated > >>>>>>>>>>>>>> enough to build over the next few years. At the end of the day, > >>>>>>>>> Apache > >>>>>>>>>>>>>> projects are do-ocracies. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> - Wes > >>>>>>>>>>>>>> On Fri, Dec 7, 2018 at 6:22 AM Andy Grove < > >>>>>>> andygrov...@gmail.com> > >>>>>>>>> wrote: > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> I've added one PR to the list ( > >>>>>>>>> https://github.com/apache/arrow/pull/3119 > >>>>>>>>>>>>>> ) > >>>>>>>>>>>>>>> to update the project to use Rust 2018 Edition. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> I'm also considering removing one PR from the list and would > >>>>>>> like > >>>>>>>>> to get > >>>>>>>>>>>>>>> opinions here. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> I have a PR (https://github.com/apache/arrow/pull/3033) to > >>>>>>> add > >>>>>>>>> some > >>>>>>>>>>>>>> basic > >>>>>>>>>>>>>>> math and comparison operators to primitive arrays. These are > >>>>>>> baby > >>>>>>>>> steps > >>>>>>>>>>>>>>> towards implementing more query execution capabilities such as > >>>>>>>>>>>>>> projection, > >>>>>>>>>>>>>>> selection, etc but Chao made a good point that other Rust > >>>>>>>>> implementations > >>>>>>>>>>>>>>> don't have these kind of capabilities and I am now wondering > >>>>>>> if > >>>>>>>>> this is a > >>>>>>>>>>>>>>> distraction. We already have Gandiva and the new efforts in > >>>>>>> Ursa > >>>>>>>>> labs and > >>>>>>>>>>>>>>> it would probably make more sense to look at having Rust > >>>>>>> bindings > >>>>>>>>> for the > >>>>>>>>>>>>>>> query execution capabilities there rather than having a > >>>>>>> competing > >>>>>>>>> (and > >>>>>>>>>>>>>> less > >>>>>>>>>>>>>>> capable) implementation in Rust. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Thoughts? > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Andy. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> On Thu, Dec 6, 2018 at 8:42 PM paddy horan < > >>>>>>>>> paddyho...@hotmail.com> > >>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Other than Andy’s PR below I’m going to try and find time to > >>>>>>>>> work on > >>>>>>>>>>>>>>>> ARROW-3827, I’ll bump it 0.13 if I can’t find the time early > >>>>>>>>> next week. > >>>>>>>>>>>>>>>> There is nothing else in the 0.12 backlog for Rust. It > >>>>>>> would be > >>>>>>>>> nice > >>>>>>>>>>>>>> to > >>>>>>>>>>>>>>>> get the parquet merge in though. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Paddy > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> ________________________________ > >>>>>>>>>>>>>>>> From: Andy Grove <andygrov...@gmail.com> > >>>>>>>>>>>>>>>> Sent: Thursday, December 6, 2018 10:20:48 AM > >>>>>>>>>>>>>>>> To: dev@arrow.apache.org > >>>>>>>>>>>>>>>> Subject: Re: Timeline for Arrow 0.12.0 release > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> I have PRs pending for all the Rust issues that I want to get > >>>>>>>>> into > >>>>>>>>>>>>>> 0.12.0 > >>>>>>>>>>>>>>>> and would appreciate some reviews so I can go ahead and > >>>>>>> merge: > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> https://github.com/apache/arrow/pull/3033 (covers > >>>>>>> ARROW-3880 and > >>>>>>>>>>>>>>>> ARROW-3881 > >>>>>>>>>>>>>>>> - add math and comparison operations to primitive arrays) > >>>>>>>>>>>>>>>> https://github.com/apache/arrow/pull/3096 (ARROW-3885 - Rust > >>>>>>>>> release > >>>>>>>>>>>>>>>> process) > >>>>>>>>>>>>>>>> https://github.com/apache/arrow/pull/3111 (ARROW-3838 - CSV > >>>>>>>>> Writer) > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> With these in place I plan on writing a tutorial for reading > >>>>>>> a > >>>>>>>>> CSV > >>>>>>>>>>>>>> file, > >>>>>>>>>>>>>>>> performing some operations on primitive arrays and writing > >>>>>>> the > >>>>>>>>> output > >>>>>>>>>>>>>> to a > >>>>>>>>>>>>>>>> new CSV file. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> I am deferring ARROW-3882 (casting for primitive arrays) to > >>>>>>>>> 0.13.0 > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Thanks, > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Andy. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> On Tue, Dec 4, 2018 at 7:57 PM Andy Grove < > >>>>>>> andygrov...@gmail.com > >>>>>>>>>> > >>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> I'd love to tackle the three related issues for supporting > >>>>>>>>> simple > >>>>>>>>>>>>>>>>> math/comparison operations on primitive arrays and casting > >>>>>>>>> primitive > >>>>>>>>>>>>>>>> arrays > >>>>>>>>>>>>>>>>> but since the change to use Rust specialization feature I'm > >>>>>>> a > >>>>>>>>> bit > >>>>>>>>>>>>>> stuck > >>>>>>>>>>>>>>>> and > >>>>>>>>>>>>>>>>> need some assistance applying the math operations to the > >>>>>>> numeric > >>>>>>>>>>>>>> types > >>>>>>>>>>>>>>>> and > >>>>>>>>>>>>>>>>> not the boolean primitives. I have added a comment to > >>>>>>>>>>>>>>>>> https://github.com/apache/arrow/pull/3033 ... if I can get > >>>>>>> help > >>>>>>>>>>>>>> solving > >>>>>>>>>>>>>>>>> for this PR then I should be able to handle the others. I'll > >>>>>>>>> also do > >>>>>>>>>>>>>> some > >>>>>>>>>>>>>>>>> research and try and figure this out myself. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Andy. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> On Tue, Dec 4, 2018 at 7:03 PM Wes McKinney < > >>>>>>>>> wesmck...@gmail.com> > >>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Andy, Paddy, or other Rust developers -- could you review > >>>>>>> the 6 > >>>>>>>>>>>>>> issues > >>>>>>>>>>>>>>>>>> in TODO in the 0.12 backlog and either assign them or move > >>>>>>>>> them to > >>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>> next release if they aren't going to be completed this > >>>>>>> week or > >>>>>>>>> next? > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> On Fri, Nov 30, 2018 at 4:34 PM Wes McKinney < > >>>>>>>>> wesmck...@gmail.com> > >>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> hi folks, > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Tomorrow is December 1. The last major Arrow release > >>>>>>> (0.11.0) > >>>>>>>>> took > >>>>>>>>>>>>>>>>>>> place on October 8. Given how much work has happened in > >>>>>>> the > >>>>>>>>>>>>>> project in > >>>>>>>>>>>>>>>>>>> the last ~2 months, I think it would be great to complete > >>>>>>> the > >>>>>>>>> next > >>>>>>>>>>>>>>>>>>> major release before the end-of-year holidays set in. > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> I've been curating the JIRA backlog the last couple of > >>>>>>> weeks, > >>>>>>>>> and > >>>>>>>>>>>>>> have > >>>>>>>>>>>>>>>>>>> just created a 0.12.0 release wiki page to help us stay > >>>>>>>>> organized > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>> > >>>> https://cwiki.apache.org/confluence/display/ARROW/Arrow+0.12.0+Release > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Given that there are only 3 full working weeks between > >>>>>>> now and > >>>>>>>>>>>>>>>>>>> Christmas, I think we should be in position to cut a > >>>>>>> release > >>>>>>>>> by > >>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>> end of the week of December 10, i.e. by Friday December > >>>>>>> 14. > >>>>>>>>> Not > >>>>>>>>>>>>>> all of > >>>>>>>>>>>>>>>>>>> the TODO issues have to be completed to make the release, > >>>>>>> but > >>>>>>>>> it > >>>>>>>>>>>>>> would > >>>>>>>>>>>>>>>>>>> be good to push to complete as much as possible. Please > >>>>>>> help > >>>>>>>>> by > >>>>>>>>>>>>>>>>>>> reviewing the backlog, and if possible, assigning issues > >>>>>>> to > >>>>>>>>>>>>>> yourself > >>>>>>>>>>>>>>>>>>> that you'd like to pursue in the next 2 weeks. > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Let me know if this sounds reasonable, or any concerns. > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Thanks > >>>>>>>>>>>>>>>>>>> Wes > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>> > >>>
