Unfortunately, we aren't quite out of the woods yet. https://issues.apache.org/jira/browse/ARROW-2318
Robert or Philipp, could you please advise on this issue? I think if we're learning anything from this experience, we need to work on our continuous integration and delivery. We shouldn't be finding out about all these problems at the least minute -- the master branch should be kept in a releasable state every day (i.e. the release verification script passes on a snapshot of master) On Thu, Mar 15, 2018 at 2:06 AM, Kouhei Sutou <k...@clear-code.com> wrote: > https://github.com/apache/arrow/pull/1752 will solve this. > > -- > kou > > In <20180315.135146.1891617944364686374....@clear-code.com> > "Re: Working towards getting 0.9.0 release candidate up next week" on Thu, > 15 Mar 2018 13:51:46 +0900 (JST), > Kouhei Sutou <k...@clear-code.com> wrote: > >> Sure. Please wait for a while. >> >> -- >> kou >> >> In <CAJPUwMCHqVSbJp5xVDXZV2qY=n5DVPqH=miaetgq_d8c5on...@mail.gmail.com> >> "Re: Working towards getting 0.9.0 release candidate up next week" on Thu, >> 15 Mar 2018 00:49:02 -0400, >> Wes McKinney <wesmck...@gmail.com> wrote: >> >>> I ran into another problem: >>> >>> https://issues.apache.org/jira/browse/ARROW-2313 >>> >>> @Kou, could you have a look at this? Thanks >>> >>> On Wed, Mar 14, 2018 at 10:36 PM, Wes McKinney <wesmck...@gmail.com> wrote: >>>> Thanks Paul! This seems to have done the trick. I will get the RC out >>>> later this evening >>>> >>>> On Wed, Mar 14, 2018 at 9:50 PM, Paul Taylor <ptay...@apache.org> wrote: >>>>> This should fix it: https://github.com/apache/arrow/pull/1751 >>>>> <https://github.com/apache/arrow/pull/1751> >>>>> >>>>>> On Mar 14, 2018, at 6:42 PM, Wes McKinney <wesmck...@gmail.com> wrote: >>>>>> >>>>>> Last item https://issues.apache.org/jira/browse/ARROW-2312 >>>>>> >>>>>> I can start the release vote as soon as we have the release >>>>>> verification script working again >>>>>> >>>>>> On Wed, Mar 14, 2018 at 2:28 PM, Wes McKinney <wesmck...@gmail.com> >>>>>> wrote: >>>>>>> OK, patch is up for ARROW-2307: >>>>>>> https://github.com/apache/arrow/pull/1747. Once that is reviewed and >>>>>>> merged I will start the release vote >>>>>>> >>>>>>> On Wed, Mar 14, 2018 at 10:18 AM, Wes McKinney <wesmck...@gmail.com> >>>>>>> wrote: >>>>>>>> I'm going to have a look at ARROW-2307 to see if it's an easy fix. If >>>>>>>> not, I will go ahead with the RC >>>>>>>> >>>>>>>> On Wed, Mar 14, 2018 at 7:36 AM, Uwe L. Korn <uw...@xhochy.com> wrote: >>>>>>>>> The mentioned bugfixes were merged and I also tested the >>>>>>>>> Arrow<->Parquet-cpp as well as the Arrow<->Dask(parquet) integration. >>>>>>>>> Both seem to work fine. From my side it looks like we're ready to >>>>>>>>> make an RC. >>>>>>>>> >>>>>>>>> Uwe >>>>>>>>> >>>>>>>>> On Wed, Mar 14, 2018, at 5:07 AM, Wes McKinney wrote: >>>>>>>>>> I fixed these bugs >>>>>>>>>> >>>>>>>>>> https://github.com/apache/arrow/pull/1742 >>>>>>>>>> https://github.com/apache/arrow/pull/1743 >>>>>>>>>> >>>>>>>>>> As soon as these patches go in, we can cut the RC0. I can do this as >>>>>>>>>> soon as tomorrow (Wednesday) morning >>>>>>>>>> >>>>>>>>>> On Tue, Mar 13, 2018 at 5:00 PM, Wes McKinney <wesmck...@gmail.com> >>>>>>>>>> wrote: >>>>>>>>>>> I found 2 bugs -- ARROW-2304 and ARROW-2306 -- while doing some >>>>>>>>>>> final >>>>>>>>>>> testing of master (stuff we haven't been testing in CI -- we >>>>>>>>>>> _really_ >>>>>>>>>>> need to set up nightly CI jobs for more time consuming tests, like >>>>>>>>>>> HDFS, that we want to test periodically but perhaps not on every >>>>>>>>>>> commit). I'm going to see if these are easy to fix >>>>>>>>>>> >>>>>>>>>>> On Tue, Mar 13, 2018 at 11:21 AM, Wes McKinney >>>>>>>>>>> <wesmck...@gmail.com> wrote: >>>>>>>>>>>> I fixed ARROW-2227 in https://github.com/apache/arrow/pull/1740, >>>>>>>>>>>> so if >>>>>>>>>>>> someone could review that would be great. >>>>>>>>>>>> >>>>>>>>>>>> I am going to make sure the C++/Python HDFS tests run locally, then >>>>>>>>>>>> after the patch above is merged we should be good to make the RC. >>>>>>>>>>>> >>>>>>>>>>>> Note to other onlookers -- please feel free to keep writing new >>>>>>>>>>>> patches not mentioned here. They may just have to go in to the next >>>>>>>>>>>> release >>>>>>>>>>>> >>>>>>>>>>>> Thanks >>>>>>>>>>>> Wes >>>>>>>>>>>> >>>>>>>>>>>> On Tue, Mar 13, 2018 at 12:56 AM, Wes McKinney >>>>>>>>>>>> <wesmck...@gmail.com> wrote: >>>>>>>>>>>>> Things are looking pretty good. I'm waiting on a build for >>>>>>>>>>>>> ARROW-1643 >>>>>>>>>>>>> to go in, and there's a last blocker bug (ARROW-2227) that we >>>>>>>>>>>>> ought to >>>>>>>>>>>>> try to fix tomorrow before cutting the RC. I can start the vote >>>>>>>>>>>>> after >>>>>>>>>>>>> that if nothing else comes up >>>>>>>>>>>>> >>>>>>>>>>>>> I created ARROW-2300 in the course of trying to test ARROW-1643. I >>>>>>>>>>>>> have an alternative way to run the HDFS tests (since we aren't >>>>>>>>>>>>> running >>>>>>>>>>>>> them in Travis CI) so most likely I will try out the HDFS tests >>>>>>>>>>>>> and >>>>>>>>>>>>> then move ARROW-2300 to the next release milestone. >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks >>>>>>>>>>>>> Wes >>>>>>>>>>>>> >>>>>>>>>>>>> On Mon, Mar 12, 2018 at 12:03 AM, Wes McKinney >>>>>>>>>>>>> <wesmck...@gmail.com> wrote: >>>>>>>>>>>>>> I've done a pass over the remaining JIRAs -- I think we're going >>>>>>>>>>>>>> to >>>>>>>>>>>>>> need at least another full day to get things buttoned up, so I >>>>>>>>>>>>>> would >>>>>>>>>>>>>> say we're looking at an RC on Tuesday. >>>>>>>>>>>>>> >>>>>>>>>>>>>> In progress: >>>>>>>>>>>>>> - ARROW-1425: Patch up, needs some editing, may be moved to >>>>>>>>>>>>>> 0.10.0 >>>>>>>>>>>>>> - ARROW-2282: Patch up, needs some test cases >>>>>>>>>>>>>> - ARROW-1974: Patch in review in apache/parquet-cpp -- may want >>>>>>>>>>>>>> to >>>>>>>>>>>>>> move this JIRA to the Parquet project >>>>>>>>>>>>>> - ARROW-2122: Patch to be reviewed >>>>>>>>>>>>>> - ARROW-2135: Patch to be merged >>>>>>>>>>>>>> >>>>>>>>>>>>>> TODO >>>>>>>>>>>>>> - ARROW-2082: Parquet segfault <- to be investigated, may be >>>>>>>>>>>>>> moved to 0.10.0 >>>>>>>>>>>>>> - ARROW-2118: Fix rough edge with reading length-0 files >>>>>>>>>>>>>> - ARROW-2227: Bug with creating chunked arrays in >>>>>>>>>>>>>> Table.from_pandas >>>>>>>>>>>>>> - ARROW-2292: Deprecation / renaming a Python method >>>>>>>>>>>>>> >>>>>>>>>>>>>> Out of the two items in TODO, ARROW-2118 and ARROW-2292 are small >>>>>>>>>>>>>> matters, so I will take care of them. ARROW-2227 may not be >>>>>>>>>>>>>> fixable >>>>>>>>>>>>>> within ~1 day but it would be useful to have a diagnosis in case >>>>>>>>>>>>>> the >>>>>>>>>>>>>> fix is easy -- multiple users hit this bug. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>> Wes >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Thu, Mar 8, 2018 at 8:47 PM, Kouhei Sutou >>>>>>>>>>>>>> <k...@clear-code.com> wrote: >>>>>>>>>>>>>>> Thanks! >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> kou >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> In >>>>>>>>>>>>>>> <cajpuwmbsxxw9-8vrj3xwud_p3y7sosdmc6xc53pkrjusewm...@mail.gmail.com> >>>>>>>>>>>>>>> "Re: Working towards getting 0.9.0 release candidate up next >>>>>>>>>>>>>>> week" on Thu, 8 Mar 2018 20:44:14 -0500, >>>>>>>>>>>>>>> Wes McKinney <wesmck...@gmail.com> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> hi Kou -- yes, I think this is a good idea. It will require a >>>>>>>>>>>>>>>> little >>>>>>>>>>>>>>>> bit of work to be able to produce a viable standalone source >>>>>>>>>>>>>>>> tarball. >>>>>>>>>>>>>>>> Between Uwe, Phillip, Antoine, and I, we should be able to >>>>>>>>>>>>>>>> come up >>>>>>>>>>>>>>>> with a plan to do this >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> - Wes >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Thu, Mar 8, 2018 at 8:33 PM, Kouhei Sutou >>>>>>>>>>>>>>>> <k...@clear-code.com> wrote: >>>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> - Updating pip packages for C++ and Python >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Can we try adding PyArrow source package to PyPI at the >>>>>>>>>>>>>>>>> 0.9.0? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I want to install PyArrow with Arrow C++ installed by .deb >>>>>>>>>>>>>>>>> or .rpm. I want to use both Red Arrow (Ruby bindings) and >>>>>>>>>>>>>>>>> PyArrow in the same process via PyCall (Ruby library >>>>>>>>>>>>>>>>> to integrate with Python). In the case, I need to use the >>>>>>>>>>>>>>>>> same Arrow C++ in both Red Arrow and PyArrow. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Now, there are only binary packages for PyArrow at >>>>>>>>>>>>>>>>> https://pypi.python.org/pypi/pyarrow . If there is a source >>>>>>>>>>>>>>>>> package for PyArrow at PyPI, I can install PyArrow with >>>>>>>>>>>>>>>>> Arrow C++ installed by .deb or .rpm by "pip --no-binary >>>>>>>>>>>>>>>>> pyarrow". >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Red Arrow can also use Arrow C++ installed by .deb or .rpm. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>> kou >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> In >>>>>>>>>>>>>>>>> <cakhw1aqgv7sn70jewe5qmw00yevx_xg_zjinywgbo00+c4k...@mail.gmail.com> >>>>>>>>>>>>>>>>> "Re: Working towards getting 0.9.0 release candidate up next >>>>>>>>>>>>>>>>> week" on Thu, 8 Mar 2018 11:25:32 -0800, >>>>>>>>>>>>>>>>> Siddharth Teotia <siddha...@dremio.com> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> All, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> I plan to get RC out over the weekend or early Monday. Is >>>>>>>>>>>>>>>>>> that fine with >>>>>>>>>>>>>>>>>> everybody? >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> We have 6 items in progress -- >>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/projects/ARROW/versions/12341707#release-report-tab-body. >>>>>>>>>>>>>>>>>> How do people feel about completing these JIRAs by tomorrow? >>>>>>>>>>>>>>>>>> I am >>>>>>>>>>>>>>>>>> completely fine with deferring the RC to early next week >>>>>>>>>>>>>>>>>> (Mon/Tue/Wed) if >>>>>>>>>>>>>>>>>> necessary. Just looking for consensus. Also, I suggest that >>>>>>>>>>>>>>>>>> we defer the >>>>>>>>>>>>>>>>>> ones with TODO status. I will do it later today unless I >>>>>>>>>>>>>>>>>> hear otherwise. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> I was wondering if anyone else is interested in >>>>>>>>>>>>>>>>>> collaborating for the >>>>>>>>>>>>>>>>>> post-release tasks. As per >>>>>>>>>>>>>>>>>> https://github.com/apache/arrow/blob/master/dev/release/RELEASE_MANAGEMENT.md, >>>>>>>>>>>>>>>>>> following are the high level post-release tasks. Please let >>>>>>>>>>>>>>>>>> me know if you >>>>>>>>>>>>>>>>>> would like to take up something. I have written my name >>>>>>>>>>>>>>>>>> against some of >>>>>>>>>>>>>>>>>> them. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> - Updating the Arrow Website (Sidd) >>>>>>>>>>>>>>>>>> - Uploading release artifacts to SVN -- looks like PMC >>>>>>>>>>>>>>>>>> karma is needed >>>>>>>>>>>>>>>>>> to do this >>>>>>>>>>>>>>>>>> - Announcing release (Sidd) >>>>>>>>>>>>>>>>>> - Updating website with new API documentation (Sidd) >>>>>>>>>>>>>>>>>> - Updating pip packages for C++ and Python >>>>>>>>>>>>>>>>>> - Updating conda packages for C++ and Python (Sidd) >>>>>>>>>>>>>>>>>> - Updating Java Maven artifacts in Maven central (Sidd) >>>>>>>>>>>>>>>>>> - Release blog post >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> If anything is missing, please add to the above list. It >>>>>>>>>>>>>>>>>> will be helpful >>>>>>>>>>>>>>>>>> for tracking. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>> Sidd >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Sun, Mar 4, 2018 at 12:34 PM, Wes McKinney >>>>>>>>>>>>>>>>>> <wesmck...@gmail.com> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> hey Sidd, >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> The Python backlog is still in pretty rough shape. I'd like >>>>>>>>>>>>>>>>>>> to see if >>>>>>>>>>>>>>>>>>> we can make an RC by Friday but if not we can defer to >>>>>>>>>>>>>>>>>>> Monday/Tuesday >>>>>>>>>>>>>>>>>>> the following week (3/12 or 13). I will trim as much as >>>>>>>>>>>>>>>>>>> possible out >>>>>>>>>>>>>>>>>>> of the current backlog to get things down to the essential >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> - Wes >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> On Sun, Feb 25, 2018 at 11:58 AM, Siddharth Teotia >>>>>>>>>>>>>>>>>>> <siddha...@dremio.com> >>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>> Sounds good. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>>>>>>>> Sidd >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> On Feb 24, 2018 6:24 PM, "Wes McKinney" >>>>>>>>>>>>>>>>>>>> <wesmck...@gmail.com> wrote: >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Hi Sidd, >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> I think we have too many bugs to make an RC this coming >>>>>>>>>>>>>>>>>>>> week. I suggest >>>>>>>>>>>>>>>>>>> we >>>>>>>>>>>>>>>>>>>> defer to the following week. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>>>>>>>> Wes >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> On Feb 24, 2018 7:09 PM, "Siddharth Teotia" >>>>>>>>>>>>>>>>>>>> <siddha...@dremio.com> >>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Hi All, >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> We currently have 10 issues in progress and PRs are >>>>>>>>>>>>>>>>>>>> available for 8 of >>>>>>>>>>>>>>>>>>>> them. In interest of getting a release candidate next >>>>>>>>>>>>>>>>>>>> week, I would >>>>>>>>>>>>>>>>>>> request >>>>>>>>>>>>>>>>>>>> people to review PRs as soon as they can to help make >>>>>>>>>>>>>>>>>>>> progress and close >>>>>>>>>>>>>>>>>>>> out as many JIRAs as we can. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> There are 32 issues in TODO list and 25 of them are not >>>>>>>>>>>>>>>>>>>> yet assigned. I >>>>>>>>>>>>>>>>>>> am >>>>>>>>>>>>>>>>>>>> planning to defer some of the unassigned ones later today >>>>>>>>>>>>>>>>>>>> or tomorrow. It >>>>>>>>>>>>>>>>>>>> would be good to soon grab/assign the issues that people >>>>>>>>>>>>>>>>>>>> want to be fixed >>>>>>>>>>>>>>>>>>>> for 0.9.0. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Here is the link to backlog: >>>>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/projects/ARROW/versions/12341707 >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>>>> Sidd >>>>>>>>>>>>>>>>>>> >>>>>
