Should We aim for Friday (2019-01-11) or Thursday (2019-01-10)? On Wed, Jan 9, 2019 at 11:43 AM Antoine Pitrou <anto...@python.org> wrote:
> > For the record, none of these issues seem release-critical to me, apart > from the RAT issue and perhaps the Windows git-describe issue. Though > perhaps the release manager is best placed to evaluate them. > > Regards > > Antoine. > > > Le 08/01/2019 à 23:07, Wes McKinney a écrit : > > Well, a mid-week release candidate isn't looking too likely. I've > > spent most of today working on the Gandiva Windows build and haven't > > been able to do much 0.12 backlog items yesterday or today so far. > > > > Krisztian -- is your GPG key in KEYS yet? If not, you cannot cut a > release. > > > > Out of the remaining issues in the backlog > > https://issues.apache.org/jira/projects/ARROW/versions/12343858 > > > > Python issues > > > > * ARROW-2038: small comment to be addressed, then merged. Someone > > please create follow up JIRA about CI for S3 > > * ARROW-2298: I just nixed this for 0.12. There is a patch up but it > > needs more work > > * ARROW-2659 and ARROW-2860 are the same issue, I think. We should > > fix as it has impacted many users and been reported many times > > * ARROW-3344: Recurring Plasma test failure on Ubuntu 14.04. I will > > take a look and see if the fix is difficult or not > > * ARROW-3428: Close to merge ready, I will review and make sure all is > good > > * ARROW-4138: Windows rough edge, I will look > > * ARROW-3916: There is a patch here; can someone look? Given the > > overlap between Arrow and Parquet users it would be good to fix this > > * ARROW-4181: Failing large_memory test -- can someone look (and also > > see if any other large_memory tests are failing)? > > > > Rust items: can be merged but need not block release > > > > * ARROW-4040 > > * ARROW-4193 > > > > The other issues > > > > * ARROW-3578: The release manager will need to be careful to see what > > happens with RAT in the build > > * ARROW-4199: patch available > > * ARROW-854: To be merged with experimental designation on green build > > * ARROW-4197: Emscripten issues with C++. Would be good to merge fix > > in time for release > > > > I would say we should close the backlog by end-of-week at latest and > > move forward with the release > > > > Thanks > > Wes > > > > On Mon, Jan 7, 2019 at 10:08 AM Krisztián Szűcs > > <szucs.kriszt...@gmail.com> wrote: > >> > >> Cool. I'm going to help with the python issues right after fixing the > spark > >> integration tests. > >> > >> On Mon, Jan 7, 2019 at 4:43 PM Wes McKinney <wesmck...@gmail.com> > wrote: > >> > >>> Great, thank you! > >>> > >>> There are 20 0.12 issues Open or In-Progress, I'm going to tackle a > >>> couple more Python things today and tomorrow, but let's see where we > >>> stand by Wednesday or so and decide when to cut the release > >>> > >>> On Mon, Jan 7, 2019 at 4:38 AM Krisztián Szűcs > >>> <szucs.kriszt...@gmail.com> wrote: > >>>> > >>>> Hey! > >>>> > >>>> On Fri, Jan 4, 2019 at 10:31 PM Wes McKinney <wesmck...@gmail.com> > >>> wrote: > >>>> > >>>>> hi all, > >>>>> > >>>>> We should try to cut a release candidate for 0.12 as soon as > >>>>> practical. Since we're just coming off the holidays, it would be good > >>>>> to work for a few more business days to close out as many outstanding > >>>>> patches as possible, and be in position to start a vote sometime next > >>>>> week. > >>>>> > >>>>> There's a bunch of Python bugs in the backlog still -- if anyone can > >>>>> pick up one or two of these it would be a help > >>>>> > >>>>> Would someone (Krisztian or Antoine maybe?) like to be the release > >>> manager? > >>>>> > >>>> Yes, I volunteer :) > >>>> > >>>>> > >>>>> Thanks > >>>>> > >>>>> On Wed, Dec 12, 2018 at 8:15 AM Wes McKinney <wesmck...@gmail.com> > >>> wrote: > >>>>>> > >>>>>> I agree that we should aim for time-based releases. Let's discuss a > >>>>>> time-based release schedule (my preference would be ~every 2 months) > >>>>>> for 2019 after we get 0.12 out. > >>>>>> On Wed, Dec 12, 2018 at 3:15 AM Antoine Pitrou <anto...@python.org> > >>>>> wrote: > >>>>>>> > >>>>>>> > >>>>>>> I think we should aim for time-based releases in general (rather > >>> than a > >>>>>>> specific set of features), but delaying this one sounds good to me. > >>>>>>> > >>>>>>> Regards > >>>>>>> > >>>>>>> Antoine. > >>>>>>> > >>>>>>> > >>>>>>> Le 12/12/2018 à 01:34, Wes McKinney a écrit : > >>>>>>>> hi all, > >>>>>>>> > >>>>>>>> I'm looking at the 0.12 backlog and I am not too comfortable > >>> with the > >>>>>>>> things that would have to be cut to get a release out next week. > >>>>>>>> Additionally, not a lot of developers are going to be working the > >>>>> week > >>>>>>>> of December 24 because of the Christmas and New Year's holidays, > >>> so > >>>>>>>> even if we did release, it might not get seen by a lot of people > >>>>> until > >>>>>>>> after the New Year. > >>>>>>>> > >>>>>>>> Based on this, I would suggest we push to complete as much work > >>> as > >>>>>>>> possible (from the 0.12 backlog and beyond) by the end of the > >>> year, > >>>>>>>> and release as soon as possible in 2019. Of course, anyone is > >>> welcome > >>>>>>>> to contribute work that is not found in the 0.12 milestone =) > >>>>>>>> > >>>>>>>> Any objections? > >>>>>>>> > >>>>>>>> Thanks > >>>>>>>> Wes > >>>>>>>> On Mon, Dec 10, 2018 at 8:04 AM Andy Grove < > >>> andygrov...@gmail.com> > >>>>> wrote: > >>>>>>>>> > >>>>>>>>> Cool. I will continue to add primitive operations but I am now > >>>>> adding this > >>>>>>>>> in a separate source file to keep it separate from the core > >>> array > >>>>> code. > >>>>>>>>> > >>>>>>>>> I'm not sure how important it will be to support Rust data > >>> sources > >>>>> with > >>>>>>>>> Gandiva. I can see that each language should be able to > >>> construct > >>>>> the > >>>>>>>>> logical query plan to submit to Gandiva and let Gandiva handle > >>>>> execution. I > >>>>>>>>> think the more interesting part is how do we support > >>>>> language-specific > >>>>>>>>> lambda functions as part of that logical query plan. Maybe it is > >>>>> possible > >>>>>>>>> to compile the lambda down to LLVM (I haven't started learning > >>>>> about LLVM > >>>>>>>>> in detail yet so this is wild speculation on my part). Another > >>>>> option is > >>>>>>>>> for Gandiva to support calling into shared libraries and that > >>> maybe > >>>>> is > >>>>>>>>> simpler for languages that support building C-native shared > >>>>> libraries (Rust > >>>>>>>>> supports this with zero overhead). > >>>>>>>>> > >>>>>>>>> Andy. > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> On Sun, Dec 9, 2018 at 11:42 AM Wes McKinney < > >>> wesmck...@gmail.com> > >>>>> wrote: > >>>>>>>>> > >>>>>>>>>> hi Andy, > >>>>>>>>>> > >>>>>>>>>> I can see an argument for having some basic native function > >>> kernel > >>>>>>>>>> support in Rust. One of the things that Gandiva has begun is a > >>>>>>>>>> Protobuf-based serialized representation representation of > >>>>> projection > >>>>>>>>>> and filter expressions. In the long run I would like to see a > >>> more > >>>>>>>>>> complete relational algebra / logical query plan that can be > >>>>> submitted > >>>>>>>>>> for execution. There's complexities, though, such as bridging > >>>>>>>>>> iteration of data sources written in Rust, say, with a query > >>> engine > >>>>>>>>>> written in C++. You would need to provide some kind of a > >>> callback > >>>>>>>>>> mechanism for the query engine to request the next chunk of a > >>>>> dataset > >>>>>>>>>> to be materialized. > >>>>>>>>>> > >>>>>>>>>> It will be interested to see what contributors will be > >>> motivated > >>>>>>>>>> enough to build over the next few years. At the end of the day, > >>>>> Apache > >>>>>>>>>> projects are do-ocracies. > >>>>>>>>>> > >>>>>>>>>> - Wes > >>>>>>>>>> On Fri, Dec 7, 2018 at 6:22 AM Andy Grove < > >>> andygrov...@gmail.com> > >>>>> wrote: > >>>>>>>>>>> > >>>>>>>>>>> I've added one PR to the list ( > >>>>> https://github.com/apache/arrow/pull/3119 > >>>>>>>>>> ) > >>>>>>>>>>> to update the project to use Rust 2018 Edition. > >>>>>>>>>>> > >>>>>>>>>>> I'm also considering removing one PR from the list and would > >>> like > >>>>> to get > >>>>>>>>>>> opinions here. > >>>>>>>>>>> > >>>>>>>>>>> I have a PR (https://github.com/apache/arrow/pull/3033) to > >>> add > >>>>> some > >>>>>>>>>> basic > >>>>>>>>>>> math and comparison operators to primitive arrays. These are > >>> baby > >>>>> steps > >>>>>>>>>>> towards implementing more query execution capabilities such as > >>>>>>>>>> projection, > >>>>>>>>>>> selection, etc but Chao made a good point that other Rust > >>>>> implementations > >>>>>>>>>>> don't have these kind of capabilities and I am now wondering > >>> if > >>>>> this is a > >>>>>>>>>>> distraction. We already have Gandiva and the new efforts in > >>> Ursa > >>>>> labs and > >>>>>>>>>>> it would probably make more sense to look at having Rust > >>> bindings > >>>>> for the > >>>>>>>>>>> query execution capabilities there rather than having a > >>> competing > >>>>> (and > >>>>>>>>>> less > >>>>>>>>>>> capable) implementation in Rust. > >>>>>>>>>>> > >>>>>>>>>>> Thoughts? > >>>>>>>>>>> > >>>>>>>>>>> Andy. > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> On Thu, Dec 6, 2018 at 8:42 PM paddy horan < > >>>>> paddyho...@hotmail.com> > >>>>>>>>>> wrote: > >>>>>>>>>>> > >>>>>>>>>>>> Other than Andy’s PR below I’m going to try and find time to > >>>>> work on > >>>>>>>>>>>> ARROW-3827, I’ll bump it 0.13 if I can’t find the time early > >>>>> next week. > >>>>>>>>>>>> There is nothing else in the 0.12 backlog for Rust. It > >>> would be > >>>>> nice > >>>>>>>>>> to > >>>>>>>>>>>> get the parquet merge in though. > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> Paddy > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> ________________________________ > >>>>>>>>>>>> From: Andy Grove <andygrov...@gmail.com> > >>>>>>>>>>>> Sent: Thursday, December 6, 2018 10:20:48 AM > >>>>>>>>>>>> To: dev@arrow.apache.org > >>>>>>>>>>>> Subject: Re: Timeline for Arrow 0.12.0 release > >>>>>>>>>>>> > >>>>>>>>>>>> I have PRs pending for all the Rust issues that I want to get > >>>>> into > >>>>>>>>>> 0.12.0 > >>>>>>>>>>>> and would appreciate some reviews so I can go ahead and > >>> merge: > >>>>>>>>>>>> > >>>>>>>>>>>> https://github.com/apache/arrow/pull/3033 (covers > >>> ARROW-3880 and > >>>>>>>>>>>> ARROW-3881 > >>>>>>>>>>>> - add math and comparison operations to primitive arrays) > >>>>>>>>>>>> https://github.com/apache/arrow/pull/3096 (ARROW-3885 - Rust > >>>>> release > >>>>>>>>>>>> process) > >>>>>>>>>>>> https://github.com/apache/arrow/pull/3111 (ARROW-3838 - CSV > >>>>> Writer) > >>>>>>>>>>>> > >>>>>>>>>>>> With these in place I plan on writing a tutorial for reading > >>> a > >>>>> CSV > >>>>>>>>>> file, > >>>>>>>>>>>> performing some operations on primitive arrays and writing > >>> the > >>>>> output > >>>>>>>>>> to a > >>>>>>>>>>>> new CSV file. > >>>>>>>>>>>> > >>>>>>>>>>>> I am deferring ARROW-3882 (casting for primitive arrays) to > >>>>> 0.13.0 > >>>>>>>>>>>> > >>>>>>>>>>>> Thanks, > >>>>>>>>>>>> > >>>>>>>>>>>> Andy. > >>>>>>>>>>>> > >>>>>>>>>>>> On Tue, Dec 4, 2018 at 7:57 PM Andy Grove < > >>> andygrov...@gmail.com > >>>>>> > >>>>>>>>>> wrote: > >>>>>>>>>>>> > >>>>>>>>>>>>> I'd love to tackle the three related issues for supporting > >>>>> simple > >>>>>>>>>>>>> math/comparison operations on primitive arrays and casting > >>>>> primitive > >>>>>>>>>>>> arrays > >>>>>>>>>>>>> but since the change to use Rust specialization feature I'm > >>> a > >>>>> bit > >>>>>>>>>> stuck > >>>>>>>>>>>> and > >>>>>>>>>>>>> need some assistance applying the math operations to the > >>> numeric > >>>>>>>>>> types > >>>>>>>>>>>> and > >>>>>>>>>>>>> not the boolean primitives. I have added a comment to > >>>>>>>>>>>>> https://github.com/apache/arrow/pull/3033 ... if I can get > >>> help > >>>>>>>>>> solving > >>>>>>>>>>>>> for this PR then I should be able to handle the others. I'll > >>>>> also do > >>>>>>>>>> some > >>>>>>>>>>>>> research and try and figure this out myself. > >>>>>>>>>>>>> > >>>>>>>>>>>>> Andy. > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> On Tue, Dec 4, 2018 at 7:03 PM Wes McKinney < > >>>>> wesmck...@gmail.com> > >>>>>>>>>> wrote: > >>>>>>>>>>>>> > >>>>>>>>>>>>>> Andy, Paddy, or other Rust developers -- could you review > >>> the 6 > >>>>>>>>>> issues > >>>>>>>>>>>>>> in TODO in the 0.12 backlog and either assign them or move > >>>>> them to > >>>>>>>>>> the > >>>>>>>>>>>>>> next release if they aren't going to be completed this > >>> week or > >>>>> next? > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> On Fri, Nov 30, 2018 at 4:34 PM Wes McKinney < > >>>>> wesmck...@gmail.com> > >>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> hi folks, > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Tomorrow is December 1. The last major Arrow release > >>> (0.11.0) > >>>>> took > >>>>>>>>>>>>>>> place on October 8. Given how much work has happened in > >>> the > >>>>>>>>>> project in > >>>>>>>>>>>>>>> the last ~2 months, I think it would be great to complete > >>> the > >>>>> next > >>>>>>>>>>>>>>> major release before the end-of-year holidays set in. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> I've been curating the JIRA backlog the last couple of > >>> weeks, > >>>>> and > >>>>>>>>>> have > >>>>>>>>>>>>>>> just created a 0.12.0 release wiki page to help us stay > >>>>> organized > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>> > https://cwiki.apache.org/confluence/display/ARROW/Arrow+0.12.0+Release > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Given that there are only 3 full working weeks between > >>> now and > >>>>>>>>>>>>>>> Christmas, I think we should be in position to cut a > >>> release > >>>>> by > >>>>>>>>>> the > >>>>>>>>>>>>>>> end of the week of December 10, i.e. by Friday December > >>> 14. > >>>>> Not > >>>>>>>>>> all of > >>>>>>>>>>>>>>> the TODO issues have to be completed to make the release, > >>> but > >>>>> it > >>>>>>>>>> would > >>>>>>>>>>>>>>> be good to push to complete as much as possible. Please > >>> help > >>>>> by > >>>>>>>>>>>>>>> reviewing the backlog, and if possible, assigning issues > >>> to > >>>>>>>>>> yourself > >>>>>>>>>>>>>>> that you'd like to pursue in the next 2 weeks. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Let me know if this sounds reasonable, or any concerns. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Thanks > >>>>>>>>>>>>>>> Wes > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>> > >>>>> > >>> >
