Re: Subject: [VOTE] Release Apache Arrow 0.15.0 - RC1
I'm seeing issues with a bad signature on Centos binaries (I had to regenerate some files here because my signature was initially missing from the dev KEYS file). Can anyone else confirm? Also, for Debian and Ubuntu I'm seeing warnings about not being able to verify the identify of my signature (I assume because I am not in the web of trust). Not sure if we care about the optics on this. Thanks, Micah On Fri, Sep 27, 2019 at 12:01 PM Wes McKinney wrote: > @David -- the macOS conda-based build is a bit finicky due to the > SDK-related issues. In any case since we can successfully produce > macOS binaries for Homebrew and wheel/conda, that's the most important > thing, though we should aim for a more refined out-of-the-box > development experience > > Not sure if > https://github.com/apache/arrow/blob/master/docs/source/developers/python.rst > contains any helpful environment configuration related information > > On Fri, Sep 27, 2019 at 9:56 AM Micah Kornfield > wrote: > > > > > > > > I checked in the key to > https://dist.apache.org/repos/dist/release/arrow/. > > > Is there something else I need to do to make it available for > verification? > > > > To answer my own question I think I missed adding them to the the dev > > branch (I just did this so hopefully they will propagate soon). > > > > On Fri, Sep 27, 2019 at 8:46 AM Micah Kornfield > > wrote: > > > > > I found that Micah's GPG key did not seem to be in KEYS and had to > > >> import it manually. > > > > > > > > > I checked in the key to > https://dist.apache.org/repos/dist/release/arrow/. > > > Is there something else I need to do to make it available for > verification? > > > > > > On Fri, Sep 27, 2019 at 7:45 AM Andy Grove > wrote: > > > > > >> +1 for Rust implementation. Tests run fine out the box. Version > numbers > > >> are correct. > > >> > > >> On Fri, Sep 27, 2019 at 7:26 AM David Li > wrote: > > >> > > >>> I ran the source verification script on MacOS 10.12 (Sierra). > > >>> > > >>> I found that Micah's GPG key did not seem to be in KEYS and had to > > >>> import it manually. > > >>> > > >>> Python failed in pyarrow/tests/test_cython.py::test_cython_api when > > >>> invoking gcc. Is this the MacOS conda toolchain issue? It works if I > > >>> add `-stdlib=libc++` to the compiler options > > >>> (https://stackoverflow.com/questions/40301619) > > >>> > > >>> Compiling pyarrow_cython_example.pyx because it changed. > > >>> [1/1] Cythonizing pyarrow_cython_example.pyx > > >>> Extension module: > > >>> > >>> 0x112c40390> > > >>> > ['/var/folders/tm/b4drxjmn7j79gtp0ppbw5qlhgn/T/arrow-0.15.0.X.O0gHnUyz/apache-arrow-0.15.0/test-miniconda/envs/arrow-test/lib/python3.6/site-packages/numpy/core/include', > > >>> > > >>> > '/private/var/folders/tm/b4drxjmn7j79gtp0ppbw5qlhgn/T/arrow-0.15.0.X.O0gHnUyz/apache-arrow-0.15.0/python/pyarrow/include'] > > >>> ['arrow', 'arrow_python'] > > >>> > > >>> > ['/private/var/folders/tm/b4drxjmn7j79gtp0ppbw5qlhgn/T/arrow-0.15.0.X.O0gHnUyz/apache-arrow-0.15.0/python/pyarrow', > > >>> > > >>> > '/var/folders/tm/b4drxjmn7j79gtp0ppbw5qlhgn/T/arrow-0.15.0.X.O0gHnUyz/install/lib'] > > >>> running build_ext > > >>> building 'pyarrow_cython_example' extension > > >>> creating build > > >>> creating build/temp.macosx-10.7-x86_64-3.6 > > >>> gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g > > >>> -fwrapv -O3 -Wall -Wstrict-prototypes > > >>> > > >>> > -I/var/folders/tm/b4drxjmn7j79gtp0ppbw5qlhgn/T/arrow-0.15.0.X.O0gHnUyz/apache-arrow-0.15.0/test-miniconda/envs/arrow-test/include > > >>> -arch x86_64 > > >>> > -I/var/folders/tm/b4drxjmn7j79gtp0ppbw5qlhgn/T/arrow-0.15.0.X.O0gHnUyz/apache-arrow-0.15.0/test-miniconda/envs/arrow-test/include > > >>> -arch x86_64 > > >>> > -I/var/folders/tm/b4drxjmn7j79gtp0ppbw5qlhgn/T/arrow-0.15.0.X.O0gHnUyz/apache-arrow-0.15.0/test-miniconda/envs/arrow-test/lib/python3.6/site-packages/numpy/core/include > > >>> > > >>> > -I/private/var/folders/tm/b4drxjmn7j79gtp0ppbw5qlhgn/T/arrow-0.15.0.X.O0gHnUyz/apache-arrow-0.15.0/python/pyarrow/include > > >>> > > >>> > -I/var/folders/tm/b4drxjmn7j79gtp0ppbw5qlhgn/T/arrow-0.15.0.X.O0gHnUyz/apache-arrow-0.15.0/test-miniconda/envs/arrow-test/include/python3.6m > > >>> -c pyarrow_cython_example.cpp -o > > >>> build/temp.macosx-10.7-x86_64-3.6/pyarrow_cython_example.o -std=c++11 > > >>> > > >>> pyarrow_cython_example.cpp:634:10: fatal error: 'unordered_map' file > not > > >>> found > > >>> #include > > >>> ^~~ > > >>> 1 error generated. > > >>> error: command 'gcc' failed with exit status 1 > > >>> > > >>> > > >>> It also failed in all the tests in > > >>> pyarrow/tests/test_plasma.py::TestPlasmaClient: > > >>> > > >>> WARNING: Logging before InitGoogleLogging() is written to STDERR > > >>> I0927 08:54:13.048408 2902139840 store.cc:1149] Allowing the Plasma > > >>> store to use up to 0.1GB of memory. > > >>> I0927 08:54:13.049072 2902139840
Re: Travis CI delays
Hello, I suggest we tackle https://jira.apache.org/jira/browse/ARROW-5801. For Rust, that would be https://jira.apache.org/jira/browse/ARROW-5809. Once ported to docker/docker-compose, it's trivial to activate github action for the same test (see https://github.com/apache/arrow/pull/5530). As I'm writing this email (4 hours after the PR was pushed), the travis checks are still not completed due to the queue. While the github action was completed after 22 minutes (https://github.com/apache/arrow/pull/5530/checks?check_run_id=239290338). François On Fri, Sep 27, 2019 at 11:22 AM Krisztián Szűcs wrote: > > On Fri, Sep 27, 2019 at 5:07 PM Andy Grove wrote: > > > Thanks Krisztian. That's very helpful. I will create a CI page on the wiki > > and add this info. > > > > Does anyone have any objections to me trying out GitHub Actions for running > > the Rust tests on PR builds? I could try this out on my own fork first. > > > I think GitHub Actions is a good idea, especially for easier build setups > like Rust > has. Go, Node as similarly straightforward, so we could decommission the > travis > counterparts hopefully speeding up the rest of the builds a bit. > > > > > On Fri, Sep 27, 2019 at 7:40 AM Krisztián Szűcs > > > > wrote: > > > > > On Fri, Sep 27, 2019 at 3:25 PM Andy Grove > > wrote: > > > > > > > I've been poking around on the Arrow website and wiki and I can't find > > > > documentation relating to CI. Do we have any documentation on how > > things > > > > work today or what the goals are? > > > > > > I don't think so. > > > > > > > For Rust builds it isn't immediately > > > > obvious why they are building on both Travis CI and Ursabot. > > > > > > > We have another thread [1], where we discuss multiple things about > > > Buildbot (ursabot). I've enabled the buildbot builder for rust because it > > > provides much quicker feedback than travis does. > > > > > > [1]: > > > > > > > > https://lists.apache.org/thread.html/02f7981176b67a12618b96b7d3b13e38b8f862e14c735dcf0ae359e0@%3Cdev.arrow.apache.org%3E > > > > > > > > > > > > > > > > > > > On Fri, Sep 27, 2019 at 6:33 AM Wes McKinney > > > wrote: > > > > > > > > > FYI, using the kibble.dev link from INFRA-18533, it seems that this > > > > > September we're using about 15% of the ASF's total Travis CI capacity > > > > > (60 concurrent workers I think) > > > > > > > > > > https://imgur.com/a/oOrbPsj > > > > > > > > > > The highest is Apache Druid (incubating) at 18%, so we are #2. > > Suffice > > > > > to say the ASF's Travis couldn't accommodate us if we had twice as > > > > > many pull requests > > > > > > > > > > On Fri, Sep 27, 2019 at 7:26 AM Wes McKinney > > > > wrote: > > > > > > > > > > > > I've been sounding the alarm bells about this for a while. We need > > to > > > > > > work to get ourselves off of Travis CI, but it is not going to be > > > > > > easy.t > > > > > > > > > > > > > > On Thu, Sep 26, 2019 at 11:42 PM Micah Kornfield < > > > > emkornfi...@gmail.com> > > > > > wrote: > > > > > > > > > > > > > > My understanding is the Travis CI queue is shared among all > > apache > > > > > > > projects, and there are few including Arrow that make heavy use > > of > > > > the > > > > > > > resources. Hence, a lot of time waiting for jobs to start. I > > > think > > > > > there > > > > > > > are some open JIRAs to finish dockerization of builds, I don't > > know > > > > the > > > > > > > current status of finding alternative CI sources though. > > > > > > > > > > > > > > On Thu, Sep 26, 2019 at 10:24 PM Andy Grove < > > andygrov...@gmail.com > > > > > > > > > wrote: > > > > > > > > > > > > > > > I know this has been discussed in the past, and I apologize for > > > not > > > > > paying > > > > > > > > attention at the time (and searching for arrow + travis in > > email > > > > > isn't very > > > > > > > > effective) but why does it take so long for our Travis CI > > builds > > > > and > > > > > are > > > > > > > > there open JIRA issues related to this? > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > Andy. > > > > > > > > > > > > > > > > > > > > > >
[jira] [Created] (ARROW-6731) [CI] [Rust] Set up Github Action to run Rust tests
Andy Grove created ARROW-6731: - Summary: [CI] [Rust] Set up Github Action to run Rust tests Key: ARROW-6731 URL: https://issues.apache.org/jira/browse/ARROW-6731 Project: Apache Arrow Issue Type: Improvement Components: CI, Rust Reporter: Andy Grove Assignee: Andy Grove Fix For: 1.0.0 Set up Github Action to run Rust tests -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ARROW-6730) [CI] Use Github Actions for "C++ with clang 7" docker image
Francois Saint-Jacques created ARROW-6730: - Summary: [CI] Use Github Actions for "C++ with clang 7" docker image Key: ARROW-6730 URL: https://issues.apache.org/jira/browse/ARROW-6730 Project: Apache Arrow Issue Type: New Feature Components: Continuous Integration Reporter: Francois Saint-Jacques -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Subject: [VOTE] Release Apache Arrow 0.15.0 - RC1
@David -- the macOS conda-based build is a bit finicky due to the SDK-related issues. In any case since we can successfully produce macOS binaries for Homebrew and wheel/conda, that's the most important thing, though we should aim for a more refined out-of-the-box development experience Not sure if https://github.com/apache/arrow/blob/master/docs/source/developers/python.rst contains any helpful environment configuration related information On Fri, Sep 27, 2019 at 9:56 AM Micah Kornfield wrote: > > > > > I checked in the key to https://dist.apache.org/repos/dist/release/arrow/. > > Is there something else I need to do to make it available for verification? > > To answer my own question I think I missed adding them to the the dev > branch (I just did this so hopefully they will propagate soon). > > On Fri, Sep 27, 2019 at 8:46 AM Micah Kornfield > wrote: > > > I found that Micah's GPG key did not seem to be in KEYS and had to > >> import it manually. > > > > > > I checked in the key to https://dist.apache.org/repos/dist/release/arrow/. > > Is there something else I need to do to make it available for verification? > > > > On Fri, Sep 27, 2019 at 7:45 AM Andy Grove wrote: > > > >> +1 for Rust implementation. Tests run fine out the box. Version numbers > >> are correct. > >> > >> On Fri, Sep 27, 2019 at 7:26 AM David Li wrote: > >> > >>> I ran the source verification script on MacOS 10.12 (Sierra). > >>> > >>> I found that Micah's GPG key did not seem to be in KEYS and had to > >>> import it manually. > >>> > >>> Python failed in pyarrow/tests/test_cython.py::test_cython_api when > >>> invoking gcc. Is this the MacOS conda toolchain issue? It works if I > >>> add `-stdlib=libc++` to the compiler options > >>> (https://stackoverflow.com/questions/40301619) > >>> > >>> Compiling pyarrow_cython_example.pyx because it changed. > >>> [1/1] Cythonizing pyarrow_cython_example.pyx > >>> Extension module: > >>> >>> 0x112c40390> > >>> ['/var/folders/tm/b4drxjmn7j79gtp0ppbw5qlhgn/T/arrow-0.15.0.X.O0gHnUyz/apache-arrow-0.15.0/test-miniconda/envs/arrow-test/lib/python3.6/site-packages/numpy/core/include', > >>> > >>> '/private/var/folders/tm/b4drxjmn7j79gtp0ppbw5qlhgn/T/arrow-0.15.0.X.O0gHnUyz/apache-arrow-0.15.0/python/pyarrow/include'] > >>> ['arrow', 'arrow_python'] > >>> > >>> ['/private/var/folders/tm/b4drxjmn7j79gtp0ppbw5qlhgn/T/arrow-0.15.0.X.O0gHnUyz/apache-arrow-0.15.0/python/pyarrow', > >>> > >>> '/var/folders/tm/b4drxjmn7j79gtp0ppbw5qlhgn/T/arrow-0.15.0.X.O0gHnUyz/install/lib'] > >>> running build_ext > >>> building 'pyarrow_cython_example' extension > >>> creating build > >>> creating build/temp.macosx-10.7-x86_64-3.6 > >>> gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g > >>> -fwrapv -O3 -Wall -Wstrict-prototypes > >>> > >>> -I/var/folders/tm/b4drxjmn7j79gtp0ppbw5qlhgn/T/arrow-0.15.0.X.O0gHnUyz/apache-arrow-0.15.0/test-miniconda/envs/arrow-test/include > >>> -arch x86_64 > >>> -I/var/folders/tm/b4drxjmn7j79gtp0ppbw5qlhgn/T/arrow-0.15.0.X.O0gHnUyz/apache-arrow-0.15.0/test-miniconda/envs/arrow-test/include > >>> -arch x86_64 > >>> -I/var/folders/tm/b4drxjmn7j79gtp0ppbw5qlhgn/T/arrow-0.15.0.X.O0gHnUyz/apache-arrow-0.15.0/test-miniconda/envs/arrow-test/lib/python3.6/site-packages/numpy/core/include > >>> > >>> -I/private/var/folders/tm/b4drxjmn7j79gtp0ppbw5qlhgn/T/arrow-0.15.0.X.O0gHnUyz/apache-arrow-0.15.0/python/pyarrow/include > >>> > >>> -I/var/folders/tm/b4drxjmn7j79gtp0ppbw5qlhgn/T/arrow-0.15.0.X.O0gHnUyz/apache-arrow-0.15.0/test-miniconda/envs/arrow-test/include/python3.6m > >>> -c pyarrow_cython_example.cpp -o > >>> build/temp.macosx-10.7-x86_64-3.6/pyarrow_cython_example.o -std=c++11 > >>> > >>> pyarrow_cython_example.cpp:634:10: fatal error: 'unordered_map' file not > >>> found > >>> #include > >>> ^~~ > >>> 1 error generated. > >>> error: command 'gcc' failed with exit status 1 > >>> > >>> > >>> It also failed in all the tests in > >>> pyarrow/tests/test_plasma.py::TestPlasmaClient: > >>> > >>> WARNING: Logging before InitGoogleLogging() is written to STDERR > >>> I0927 08:54:13.048408 2902139840 store.cc:1149] Allowing the Plasma > >>> store to use up to 0.1GB of memory. > >>> I0927 08:54:13.049072 2902139840 store.cc:1176] Starting object store > >>> with directory /tmp and huge page support disabled > >>> E0927 08:54:13.049446 2902139840 io.cc:137] Socket pathname is too long. > >>> F0927 08:54:13.049515 2902139840 store.cc:1074] Check failed: socket >= > >>> 0 > >>> *** Check failure stack trace: *** > >>> *** Aborted at 1569588853 (unix time) try "date -d @1569588853" if you > >>> are using GNU date *** > >>> PC: @0x0 (unknown) > >>> *** SIGABRT (@0x7fffa41c7d42) received by PID 61396 (TID > >>> 0x7fffacfb23c0) stack trace: *** > >>> @ 0x7fffa42a8b3a _sigtramp > >>> > >>> For this latter issue, I believe we ran into this with Flight last > >>> release.
[jira] [Created] (ARROW-6729) StlStringBuffer constructor is not zero-copy
Pasha Stetsenko created ARROW-6729: -- Summary: StlStringBuffer constructor is not zero-copy Key: ARROW-6729 URL: https://issues.apache.org/jira/browse/ARROW-6729 Project: Apache Arrow Issue Type: Improvement Components: C++ Reporter: Pasha Stetsenko Fixed in [https://github.com/apache/arrow/pull/5517] -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Travis CI delays
On Fri, Sep 27, 2019 at 5:07 PM Andy Grove wrote: > Thanks Krisztian. That's very helpful. I will create a CI page on the wiki > and add this info. > > Does anyone have any objections to me trying out GitHub Actions for running > the Rust tests on PR builds? I could try this out on my own fork first. > I think GitHub Actions is a good idea, especially for easier build setups like Rust has. Go, Node as similarly straightforward, so we could decommission the travis counterparts hopefully speeding up the rest of the builds a bit. > > On Fri, Sep 27, 2019 at 7:40 AM Krisztián Szűcs > > wrote: > > > On Fri, Sep 27, 2019 at 3:25 PM Andy Grove > wrote: > > > > > I've been poking around on the Arrow website and wiki and I can't find > > > documentation relating to CI. Do we have any documentation on how > things > > > work today or what the goals are? > > > > I don't think so. > > > > > For Rust builds it isn't immediately > > > obvious why they are building on both Travis CI and Ursabot. > > > > > We have another thread [1], where we discuss multiple things about > > Buildbot (ursabot). I've enabled the buildbot builder for rust because it > > provides much quicker feedback than travis does. > > > > [1]: > > > > > https://lists.apache.org/thread.html/02f7981176b67a12618b96b7d3b13e38b8f862e14c735dcf0ae359e0@%3Cdev.arrow.apache.org%3E > > > > > > > > > > > > > > On Fri, Sep 27, 2019 at 6:33 AM Wes McKinney > > wrote: > > > > > > > FYI, using the kibble.dev link from INFRA-18533, it seems that this > > > > September we're using about 15% of the ASF's total Travis CI capacity > > > > (60 concurrent workers I think) > > > > > > > > https://imgur.com/a/oOrbPsj > > > > > > > > The highest is Apache Druid (incubating) at 18%, so we are #2. > Suffice > > > > to say the ASF's Travis couldn't accommodate us if we had twice as > > > > many pull requests > > > > > > > > On Fri, Sep 27, 2019 at 7:26 AM Wes McKinney > > > wrote: > > > > > > > > > > I've been sounding the alarm bells about this for a while. We need > to > > > > > work to get ourselves off of Travis CI, but it is not going to be > > > > > easy.t > > > > > > > > > > > On Thu, Sep 26, 2019 at 11:42 PM Micah Kornfield < > > > emkornfi...@gmail.com> > > > > wrote: > > > > > > > > > > > > My understanding is the Travis CI queue is shared among all > apache > > > > > > projects, and there are few including Arrow that make heavy use > of > > > the > > > > > > resources. Hence, a lot of time waiting for jobs to start. I > > think > > > > there > > > > > > are some open JIRAs to finish dockerization of builds, I don't > know > > > the > > > > > > current status of finding alternative CI sources though. > > > > > > > > > > > > On Thu, Sep 26, 2019 at 10:24 PM Andy Grove < > andygrov...@gmail.com > > > > > > > wrote: > > > > > > > > > > > > > I know this has been discussed in the past, and I apologize for > > not > > > > paying > > > > > > > attention at the time (and searching for arrow + travis in > email > > > > isn't very > > > > > > > effective) but why does it take so long for our Travis CI > builds > > > and > > > > are > > > > > > > there open JIRA issues related to this? > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > Andy. > > > > > > > > > > > > > > > > >
[jira] [Created] (ARROW-6728) [C#] Support reading and writing Date32 and Date64 arrays
Eric Erhardt created ARROW-6728: --- Summary: [C#] Support reading and writing Date32 and Date64 arrays Key: ARROW-6728 URL: https://issues.apache.org/jira/browse/ARROW-6728 Project: Apache Arrow Issue Type: Bug Components: C# Reporter: Eric Erhardt The C# implementation doesn't support reading and writing Date32 and Date64 arrays. We need to add support and some tests. It looks like it is only a couple of lines to get this enabled. See [https://github.com/apache/arrow/pull/5413]. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Travis CI delays
Thanks Krisztian. That's very helpful. I will create a CI page on the wiki and add this info. Does anyone have any objections to me trying out GitHub Actions for running the Rust tests on PR builds? I could try this out on my own fork first. On Fri, Sep 27, 2019 at 7:40 AM Krisztián Szűcs wrote: > On Fri, Sep 27, 2019 at 3:25 PM Andy Grove wrote: > > > I've been poking around on the Arrow website and wiki and I can't find > > documentation relating to CI. Do we have any documentation on how things > > work today or what the goals are? > > I don't think so. > > > For Rust builds it isn't immediately > > obvious why they are building on both Travis CI and Ursabot. > > > We have another thread [1], where we discuss multiple things about > Buildbot (ursabot). I've enabled the buildbot builder for rust because it > provides much quicker feedback than travis does. > > [1]: > > https://lists.apache.org/thread.html/02f7981176b67a12618b96b7d3b13e38b8f862e14c735dcf0ae359e0@%3Cdev.arrow.apache.org%3E > > > > > > > > > On Fri, Sep 27, 2019 at 6:33 AM Wes McKinney > wrote: > > > > > FYI, using the kibble.dev link from INFRA-18533, it seems that this > > > September we're using about 15% of the ASF's total Travis CI capacity > > > (60 concurrent workers I think) > > > > > > https://imgur.com/a/oOrbPsj > > > > > > The highest is Apache Druid (incubating) at 18%, so we are #2. Suffice > > > to say the ASF's Travis couldn't accommodate us if we had twice as > > > many pull requests > > > > > > On Fri, Sep 27, 2019 at 7:26 AM Wes McKinney > > wrote: > > > > > > > > I've been sounding the alarm bells about this for a while. We need to > > > > work to get ourselves off of Travis CI, but it is not going to be > > > > easy.t > > > > > > > > On Thu, Sep 26, 2019 at 11:42 PM Micah Kornfield < > > emkornfi...@gmail.com> > > > wrote: > > > > > > > > > > My understanding is the Travis CI queue is shared among all apache > > > > > projects, and there are few including Arrow that make heavy use of > > the > > > > > resources. Hence, a lot of time waiting for jobs to start. I > think > > > there > > > > > are some open JIRAs to finish dockerization of builds, I don't know > > the > > > > > current status of finding alternative CI sources though. > > > > > > > > > > On Thu, Sep 26, 2019 at 10:24 PM Andy Grove > > > > wrote: > > > > > > > > > > > I know this has been discussed in the past, and I apologize for > not > > > paying > > > > > > attention at the time (and searching for arrow + travis in email > > > isn't very > > > > > > effective) but why does it take so long for our Travis CI builds > > and > > > are > > > > > > there open JIRA issues related to this? > > > > > > > > > > > > Thanks, > > > > > > > > > > > > Andy. > > > > > > > > > > > >
Re: Subject: [VOTE] Release Apache Arrow 0.15.0 - RC1
> > I checked in the key to https://dist.apache.org/repos/dist/release/arrow/. > Is there something else I need to do to make it available for verification? To answer my own question I think I missed adding them to the the dev branch (I just did this so hopefully they will propagate soon). On Fri, Sep 27, 2019 at 8:46 AM Micah Kornfield wrote: > I found that Micah's GPG key did not seem to be in KEYS and had to >> import it manually. > > > I checked in the key to https://dist.apache.org/repos/dist/release/arrow/. > Is there something else I need to do to make it available for verification? > > On Fri, Sep 27, 2019 at 7:45 AM Andy Grove wrote: > >> +1 for Rust implementation. Tests run fine out the box. Version numbers >> are correct. >> >> On Fri, Sep 27, 2019 at 7:26 AM David Li wrote: >> >>> I ran the source verification script on MacOS 10.12 (Sierra). >>> >>> I found that Micah's GPG key did not seem to be in KEYS and had to >>> import it manually. >>> >>> Python failed in pyarrow/tests/test_cython.py::test_cython_api when >>> invoking gcc. Is this the MacOS conda toolchain issue? It works if I >>> add `-stdlib=libc++` to the compiler options >>> (https://stackoverflow.com/questions/40301619) >>> >>> Compiling pyarrow_cython_example.pyx because it changed. >>> [1/1] Cythonizing pyarrow_cython_example.pyx >>> Extension module: >>> >> 0x112c40390> >>> ['/var/folders/tm/b4drxjmn7j79gtp0ppbw5qlhgn/T/arrow-0.15.0.X.O0gHnUyz/apache-arrow-0.15.0/test-miniconda/envs/arrow-test/lib/python3.6/site-packages/numpy/core/include', >>> >>> '/private/var/folders/tm/b4drxjmn7j79gtp0ppbw5qlhgn/T/arrow-0.15.0.X.O0gHnUyz/apache-arrow-0.15.0/python/pyarrow/include'] >>> ['arrow', 'arrow_python'] >>> >>> ['/private/var/folders/tm/b4drxjmn7j79gtp0ppbw5qlhgn/T/arrow-0.15.0.X.O0gHnUyz/apache-arrow-0.15.0/python/pyarrow', >>> >>> '/var/folders/tm/b4drxjmn7j79gtp0ppbw5qlhgn/T/arrow-0.15.0.X.O0gHnUyz/install/lib'] >>> running build_ext >>> building 'pyarrow_cython_example' extension >>> creating build >>> creating build/temp.macosx-10.7-x86_64-3.6 >>> gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g >>> -fwrapv -O3 -Wall -Wstrict-prototypes >>> >>> -I/var/folders/tm/b4drxjmn7j79gtp0ppbw5qlhgn/T/arrow-0.15.0.X.O0gHnUyz/apache-arrow-0.15.0/test-miniconda/envs/arrow-test/include >>> -arch x86_64 >>> -I/var/folders/tm/b4drxjmn7j79gtp0ppbw5qlhgn/T/arrow-0.15.0.X.O0gHnUyz/apache-arrow-0.15.0/test-miniconda/envs/arrow-test/include >>> -arch x86_64 >>> -I/var/folders/tm/b4drxjmn7j79gtp0ppbw5qlhgn/T/arrow-0.15.0.X.O0gHnUyz/apache-arrow-0.15.0/test-miniconda/envs/arrow-test/lib/python3.6/site-packages/numpy/core/include >>> >>> -I/private/var/folders/tm/b4drxjmn7j79gtp0ppbw5qlhgn/T/arrow-0.15.0.X.O0gHnUyz/apache-arrow-0.15.0/python/pyarrow/include >>> >>> -I/var/folders/tm/b4drxjmn7j79gtp0ppbw5qlhgn/T/arrow-0.15.0.X.O0gHnUyz/apache-arrow-0.15.0/test-miniconda/envs/arrow-test/include/python3.6m >>> -c pyarrow_cython_example.cpp -o >>> build/temp.macosx-10.7-x86_64-3.6/pyarrow_cython_example.o -std=c++11 >>> >>> pyarrow_cython_example.cpp:634:10: fatal error: 'unordered_map' file not >>> found >>> #include >>> ^~~ >>> 1 error generated. >>> error: command 'gcc' failed with exit status 1 >>> >>> >>> It also failed in all the tests in >>> pyarrow/tests/test_plasma.py::TestPlasmaClient: >>> >>> WARNING: Logging before InitGoogleLogging() is written to STDERR >>> I0927 08:54:13.048408 2902139840 store.cc:1149] Allowing the Plasma >>> store to use up to 0.1GB of memory. >>> I0927 08:54:13.049072 2902139840 store.cc:1176] Starting object store >>> with directory /tmp and huge page support disabled >>> E0927 08:54:13.049446 2902139840 io.cc:137] Socket pathname is too long. >>> F0927 08:54:13.049515 2902139840 store.cc:1074] Check failed: socket >= >>> 0 >>> *** Check failure stack trace: *** >>> *** Aborted at 1569588853 (unix time) try "date -d @1569588853" if you >>> are using GNU date *** >>> PC: @0x0 (unknown) >>> *** SIGABRT (@0x7fffa41c7d42) received by PID 61396 (TID >>> 0x7fffacfb23c0) stack trace: *** >>> @ 0x7fffa42a8b3a _sigtramp >>> >>> For this latter issue, I believe we ran into this with Flight last >>> release. MacOS generates very long temporary directory paths, and this >>> usually exceeds the limit on domain socket path lengths. (I also can't >>> get mktemp to respect TMPDIR.) >>> >>> Best, >>> David >>> >>> >>> On 9/27/19, Micah Kornfield wrote: >>> > Hi, >>> > >>> > >>> > I would like to propose the following release candidate (RC1) of Apache >>> > >>> > Arrow version 0.15.0. This is a release consisting of 672 >>> > >>> > resolved JIRA issues[1]. (RC0 was aborted due to release script >>> issues). >>> > >>> > >>> > This release candidate is based on commit: >>> > >>> > 77f091888dbe5a4e023a66b986a2dd474696061a [2] >>> > >>> > >>> > The source release rc1 is hosted at [3]. >>> > >>> >
Re: Subject: [VOTE] Release Apache Arrow 0.15.0 - RC1
> > I found that Micah's GPG key did not seem to be in KEYS and had to > import it manually. I checked in the key to https://dist.apache.org/repos/dist/release/arrow/. Is there something else I need to do to make it available for verification? On Fri, Sep 27, 2019 at 7:45 AM Andy Grove wrote: > +1 for Rust implementation. Tests run fine out the box. Version numbers > are correct. > > On Fri, Sep 27, 2019 at 7:26 AM David Li wrote: > >> I ran the source verification script on MacOS 10.12 (Sierra). >> >> I found that Micah's GPG key did not seem to be in KEYS and had to >> import it manually. >> >> Python failed in pyarrow/tests/test_cython.py::test_cython_api when >> invoking gcc. Is this the MacOS conda toolchain issue? It works if I >> add `-stdlib=libc++` to the compiler options >> (https://stackoverflow.com/questions/40301619) >> >> Compiling pyarrow_cython_example.pyx because it changed. >> [1/1] Cythonizing pyarrow_cython_example.pyx >> Extension module: >> > 0x112c40390> >> ['/var/folders/tm/b4drxjmn7j79gtp0ppbw5qlhgn/T/arrow-0.15.0.X.O0gHnUyz/apache-arrow-0.15.0/test-miniconda/envs/arrow-test/lib/python3.6/site-packages/numpy/core/include', >> >> '/private/var/folders/tm/b4drxjmn7j79gtp0ppbw5qlhgn/T/arrow-0.15.0.X.O0gHnUyz/apache-arrow-0.15.0/python/pyarrow/include'] >> ['arrow', 'arrow_python'] >> >> ['/private/var/folders/tm/b4drxjmn7j79gtp0ppbw5qlhgn/T/arrow-0.15.0.X.O0gHnUyz/apache-arrow-0.15.0/python/pyarrow', >> >> '/var/folders/tm/b4drxjmn7j79gtp0ppbw5qlhgn/T/arrow-0.15.0.X.O0gHnUyz/install/lib'] >> running build_ext >> building 'pyarrow_cython_example' extension >> creating build >> creating build/temp.macosx-10.7-x86_64-3.6 >> gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g >> -fwrapv -O3 -Wall -Wstrict-prototypes >> >> -I/var/folders/tm/b4drxjmn7j79gtp0ppbw5qlhgn/T/arrow-0.15.0.X.O0gHnUyz/apache-arrow-0.15.0/test-miniconda/envs/arrow-test/include >> -arch x86_64 >> -I/var/folders/tm/b4drxjmn7j79gtp0ppbw5qlhgn/T/arrow-0.15.0.X.O0gHnUyz/apache-arrow-0.15.0/test-miniconda/envs/arrow-test/include >> -arch x86_64 >> -I/var/folders/tm/b4drxjmn7j79gtp0ppbw5qlhgn/T/arrow-0.15.0.X.O0gHnUyz/apache-arrow-0.15.0/test-miniconda/envs/arrow-test/lib/python3.6/site-packages/numpy/core/include >> >> -I/private/var/folders/tm/b4drxjmn7j79gtp0ppbw5qlhgn/T/arrow-0.15.0.X.O0gHnUyz/apache-arrow-0.15.0/python/pyarrow/include >> >> -I/var/folders/tm/b4drxjmn7j79gtp0ppbw5qlhgn/T/arrow-0.15.0.X.O0gHnUyz/apache-arrow-0.15.0/test-miniconda/envs/arrow-test/include/python3.6m >> -c pyarrow_cython_example.cpp -o >> build/temp.macosx-10.7-x86_64-3.6/pyarrow_cython_example.o -std=c++11 >> >> pyarrow_cython_example.cpp:634:10: fatal error: 'unordered_map' file not >> found >> #include >> ^~~ >> 1 error generated. >> error: command 'gcc' failed with exit status 1 >> >> >> It also failed in all the tests in >> pyarrow/tests/test_plasma.py::TestPlasmaClient: >> >> WARNING: Logging before InitGoogleLogging() is written to STDERR >> I0927 08:54:13.048408 2902139840 store.cc:1149] Allowing the Plasma >> store to use up to 0.1GB of memory. >> I0927 08:54:13.049072 2902139840 store.cc:1176] Starting object store >> with directory /tmp and huge page support disabled >> E0927 08:54:13.049446 2902139840 io.cc:137] Socket pathname is too long. >> F0927 08:54:13.049515 2902139840 store.cc:1074] Check failed: socket >= 0 >> *** Check failure stack trace: *** >> *** Aborted at 1569588853 (unix time) try "date -d @1569588853" if you >> are using GNU date *** >> PC: @0x0 (unknown) >> *** SIGABRT (@0x7fffa41c7d42) received by PID 61396 (TID >> 0x7fffacfb23c0) stack trace: *** >> @ 0x7fffa42a8b3a _sigtramp >> >> For this latter issue, I believe we ran into this with Flight last >> release. MacOS generates very long temporary directory paths, and this >> usually exceeds the limit on domain socket path lengths. (I also can't >> get mktemp to respect TMPDIR.) >> >> Best, >> David >> >> >> On 9/27/19, Micah Kornfield wrote: >> > Hi, >> > >> > >> > I would like to propose the following release candidate (RC1) of Apache >> > >> > Arrow version 0.15.0. This is a release consisting of 672 >> > >> > resolved JIRA issues[1]. (RC0 was aborted due to release script issues). >> > >> > >> > This release candidate is based on commit: >> > >> > 77f091888dbe5a4e023a66b986a2dd474696061a [2] >> > >> > >> > The source release rc1 is hosted at [3]. >> > >> > The binary artifacts are hosted at [4][5][6][7]. >> > >> > The changelog is located at [8]. >> > >> > >> > Please download, verify checksums and signatures, run the unit tests, >> > >> > and vote on the release. See [9] for how to validate a release >> candidate. >> > >> > >> > The vote will be open for at least 72 hours. >> > >> > >> > [ ] +1 Release this as Apache Arrow 0.15.0 >> > >> > [ ] +0 >> > >> > [ ] -1 Do not release this as Apache Arrow 0.15.0 because..
Re: [NIGHTLY] Arrow Build Report for Job nightly-2019-09-27-0
On Fri, Sep 27, 2019 at 4:02 PM Crossbow wrote: > > Arrow Build Report for Job nightly-2019-09-27-0 > > All tasks: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-27-0 > > Failed Tasks: > - debian-stretch: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-27-0-azure-debian-stretch The build has actually succeeded, but GitHub has dropped the connection during the artifact uploading. Created a JIRA for it: https://issues.apache.org/jira/browse/ARROW-6727 > > - docker-cpp-fuzzit: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-27-0-circle-docker-cpp-fuzzit > - docker-spark-integration: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-27-0-circle-docker-spark-integration Bryan has recently fixed it, but spark hits CircleCI's timeout limitation, which hopefully will be resolved by: https://github.com/apache/arrow/pull/5485 > - gandiva-jar-osx: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-27-0-travis-gandiva-jar-osx > - docker-dask-integration: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-27-0-circle-docker-dask-integration > - wheel-manylinux1-cp27mu: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-27-0-travis-wheel-manylinux1-cp27mu Docker pull timeout issue, probably transient. > > > Succeeded Tasks: > - docker-clang-format: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-27-0-circle-docker-clang-format > - wheel-manylinux2010-cp36m: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-27-0-travis-wheel-manylinux2010-cp36m > - wheel-osx-cp35m: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-27-0-travis-wheel-osx-cp35m > - docker-r-conda: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-27-0-circle-docker-r-conda > - debian-buster: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-27-0-azure-debian-buster > - docker-r: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-27-0-circle-docker-r > - docker-turbodbc-integration: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-27-0-circle-docker-turbodbc-integration > - centos-7: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-27-0-azure-centos-7 > - conda-osx-clang-py37: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-27-0-azure-conda-osx-clang-py37 > - debian-buster-arm64: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-27-0-azure-debian-buster-arm64 > - ubuntu-disco: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-27-0-azure-ubuntu-disco > - conda-linux-gcc-py37: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-27-0-azure-conda-linux-gcc-py37 > - centos-6: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-27-0-azure-centos-6 > - docker-rust: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-27-0-circle-docker-rust > - docker-pandas-master: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-27-0-circle-docker-pandas-master > - wheel-osx-cp27m: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-27-0-travis-wheel-osx-cp27m > - wheel-osx-cp36m: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-27-0-travis-wheel-osx-cp36m > - docker-c_glib: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-27-0-circle-docker-c_glib > - conda-linux-gcc-py27: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-27-0-azure-conda-linux-gcc-py27 > - conda-win-vs2015-py36: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-27-0-azure-conda-win-vs2015-py36 > - docker-docs: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-27-0-circle-docker-docs > - docker-python-2.7-nopandas: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-27-0-circle-docker-python-2.7-nopandas > - docker-iwyu: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-27-0-circle-docker-iwyu > - docker-hdfs-integration: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-27-0-circle-docker-hdfs-integration > - ubuntu-xenial-arm64: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-27-0-azure-ubuntu-xenial-arm64 > - docker-go: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-27-0-circle-docker-go > - docker-cpp-
[jira] [Created] (ARROW-6727) [Crossbow] Improve the GitHub asset uploading prevent GitHub timeouts
Krisztian Szucs created ARROW-6727: -- Summary: [Crossbow] Improve the GitHub asset uploading prevent GitHub timeouts Key: ARROW-6727 URL: https://issues.apache.org/jira/browse/ARROW-6727 Project: Apache Arrow Issue Type: Improvement Components: Continuous Integration Reporter: Krisztian Szucs For large assets artefact uploading occasionally fails: https://dev.azure.com/ursa-labs/crossbow/_build/results?buildId=1453 Either increate the timeout in github3.py, it that is not possible then retry the upload. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[NIGHTLY] Arrow Build Report for Job nightly-2019-09-27-0
Arrow Build Report for Job nightly-2019-09-27-0 All tasks: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-27-0 Failed Tasks: - debian-stretch: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-27-0-azure-debian-stretch - docker-cpp-fuzzit: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-27-0-circle-docker-cpp-fuzzit - docker-spark-integration: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-27-0-circle-docker-spark-integration - gandiva-jar-osx: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-27-0-travis-gandiva-jar-osx - docker-dask-integration: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-27-0-circle-docker-dask-integration - wheel-manylinux1-cp27mu: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-27-0-travis-wheel-manylinux1-cp27mu Succeeded Tasks: - docker-clang-format: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-27-0-circle-docker-clang-format - wheel-manylinux2010-cp36m: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-27-0-travis-wheel-manylinux2010-cp36m - wheel-osx-cp35m: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-27-0-travis-wheel-osx-cp35m - docker-r-conda: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-27-0-circle-docker-r-conda - debian-buster: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-27-0-azure-debian-buster - docker-r: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-27-0-circle-docker-r - docker-turbodbc-integration: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-27-0-circle-docker-turbodbc-integration - centos-7: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-27-0-azure-centos-7 - conda-osx-clang-py37: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-27-0-azure-conda-osx-clang-py37 - debian-buster-arm64: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-27-0-azure-debian-buster-arm64 - ubuntu-disco: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-27-0-azure-ubuntu-disco - conda-linux-gcc-py37: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-27-0-azure-conda-linux-gcc-py37 - centos-6: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-27-0-azure-centos-6 - docker-rust: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-27-0-circle-docker-rust - docker-pandas-master: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-27-0-circle-docker-pandas-master - wheel-osx-cp27m: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-27-0-travis-wheel-osx-cp27m - wheel-osx-cp36m: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-27-0-travis-wheel-osx-cp36m - docker-c_glib: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-27-0-circle-docker-c_glib - conda-linux-gcc-py27: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-27-0-azure-conda-linux-gcc-py27 - conda-win-vs2015-py36: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-27-0-azure-conda-win-vs2015-py36 - docker-docs: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-27-0-circle-docker-docs - docker-python-2.7-nopandas: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-27-0-circle-docker-python-2.7-nopandas - docker-iwyu: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-27-0-circle-docker-iwyu - docker-hdfs-integration: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-27-0-circle-docker-hdfs-integration - ubuntu-xenial-arm64: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-27-0-azure-ubuntu-xenial-arm64 - docker-go: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-27-0-circle-docker-go - docker-cpp-cmake32: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-27-0-circle-docker-cpp-cmake32 - debian-stretch-arm64: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-27-0-azure-debian-stretch-arm64 - conda-win-vs2015-py37: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-27-0-azure-conda-win-vs2015-py37 - docker-lint: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-27-0-circle-docker-lint - wheel-win-cp36m: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-20
[jira] [Created] (ARROW-6726) [CI] Figure out how to run reliably the fuzzit task
Krisztian Szucs created ARROW-6726: -- Summary: [CI] Figure out how to run reliably the fuzzit task Key: ARROW-6726 URL: https://issues.apache.org/jira/browse/ARROW-6726 Project: Apache Arrow Issue Type: Improvement Components: Continuous Integration Reporter: Krisztian Szucs It was disabled in https://github.com/apache/arrow/pull/5528 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ARROW-6725) [CI] Disable 3rdparty fuzzit nightly builds
Krisztian Szucs created ARROW-6725: -- Summary: [CI] Disable 3rdparty fuzzit nightly builds Key: ARROW-6725 URL: https://issues.apache.org/jira/browse/ARROW-6725 Project: Apache Arrow Issue Type: Improvement Components: Continuous Integration Reporter: Krisztian Szucs Docker-cpp-fuzzit docker-compose task fails currently, probably misses parameters like version, git object id and credentials. Disable it until we have a solid solution for running them regularly. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: [NIGHTLY] Arrow Build Report for Job nightly-2019-09-26-0
Agree, I'll create a PR and an issue to revisit the fuzzit integration. On Thu, Sep 26, 2019 at 9:24 PM Wes McKinney wrote: > Should we disable the fuzzit job? This is for a third party CI-type > service, so the failure here seems like it's adding unneeded noise > > On Thu, Sep 26, 2019 at 12:31 PM Crossbow wrote: > > > > > > Arrow Build Report for Job nightly-2019-09-26-0 > > > > All tasks: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-26-0 > > > > Failed Tasks: > > - docker-spark-integration: > > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-26-0-circle-docker-spark-integration > > - docker-dask-integration: > > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-26-0-circle-docker-dask-integration > > - docker-cpp-fuzzit: > > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-26-0-circle-docker-cpp-fuzzit > > > > Succeeded Tasks: > > - wheel-win-cp36m: > > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-26-0-appveyor-wheel-win-cp36m > > - wheel-manylinux1-cp37m: > > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-26-0-travis-wheel-manylinux1-cp37m > > - homebrew-cpp-autobrew: > > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-26-0-travis-homebrew-cpp-autobrew > > - conda-win-vs2015-py37: > > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-26-0-azure-conda-win-vs2015-py37 > > - wheel-manylinux1-cp35m: > > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-26-0-travis-wheel-manylinux1-cp35m > > - docker-pandas-master: > > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-26-0-circle-docker-pandas-master > > - centos-6: > > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-26-0-azure-centos-6 > > - conda-osx-clang-py36: > > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-26-0-azure-conda-osx-clang-py36 > > - docker-turbodbc-integration: > > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-26-0-circle-docker-turbodbc-integration > > - docker-cpp-release: > > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-26-0-circle-docker-cpp-release > > - docker-docs: > > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-26-0-circle-docker-docs > > - debian-buster-arm64: > > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-26-0-azure-debian-buster-arm64 > > - debian-stretch: > > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-26-0-azure-debian-stretch > > - wheel-manylinux1-cp27mu: > > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-26-0-travis-wheel-manylinux1-cp27mu > > - docker-cpp-static-only: > > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-26-0-circle-docker-cpp-static-only > > - docker-r: > > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-26-0-circle-docker-r > > - wheel-manylinux2010-cp27m: > > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-26-0-travis-wheel-manylinux2010-cp27m > > - debian-buster: > > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-26-0-azure-debian-buster > > - wheel-manylinux1-cp27m: > > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-26-0-travis-wheel-manylinux1-cp27m > > - wheel-win-cp37m: > > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-26-0-appveyor-wheel-win-cp37m > > - ubuntu-bionic-arm64: > > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-26-0-azure-ubuntu-bionic-arm64 > > - gandiva-jar-trusty: > > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-26-0-travis-gandiva-jar-trusty > > - docker-js: > > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-26-0-circle-docker-js > > - docker-python-2.7-nopandas: > > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-26-0-circle-docker-python-2.7-nopandas > > - wheel-manylinux2010-cp37m: > > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-26-0-travis-wheel-manylinux2010-cp37m > > - wheel-win-cp35m: > > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-26-0-appveyor-wheel-win-cp35m > > - docker-cpp: > > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-26-0-circle-docker-cpp > > - conda-osx-clang-py37: > > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-26-0-azure-conda-osx-clang-py37 > > - ubuntu-d
Re: Subject: [VOTE] Release Apache Arrow 0.15.0 - RC1
+1 for Rust implementation. Tests run fine out the box. Version numbers are correct. On Fri, Sep 27, 2019 at 7:26 AM David Li wrote: > I ran the source verification script on MacOS 10.12 (Sierra). > > I found that Micah's GPG key did not seem to be in KEYS and had to > import it manually. > > Python failed in pyarrow/tests/test_cython.py::test_cython_api when > invoking gcc. Is this the MacOS conda toolchain issue? It works if I > add `-stdlib=libc++` to the compiler options > (https://stackoverflow.com/questions/40301619) > > Compiling pyarrow_cython_example.pyx because it changed. > [1/1] Cythonizing pyarrow_cython_example.pyx > Extension module: > 0x112c40390> > ['/var/folders/tm/b4drxjmn7j79gtp0ppbw5qlhgn/T/arrow-0.15.0.X.O0gHnUyz/apache-arrow-0.15.0/test-miniconda/envs/arrow-test/lib/python3.6/site-packages/numpy/core/include', > > '/private/var/folders/tm/b4drxjmn7j79gtp0ppbw5qlhgn/T/arrow-0.15.0.X.O0gHnUyz/apache-arrow-0.15.0/python/pyarrow/include'] > ['arrow', 'arrow_python'] > > ['/private/var/folders/tm/b4drxjmn7j79gtp0ppbw5qlhgn/T/arrow-0.15.0.X.O0gHnUyz/apache-arrow-0.15.0/python/pyarrow', > > '/var/folders/tm/b4drxjmn7j79gtp0ppbw5qlhgn/T/arrow-0.15.0.X.O0gHnUyz/install/lib'] > running build_ext > building 'pyarrow_cython_example' extension > creating build > creating build/temp.macosx-10.7-x86_64-3.6 > gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g > -fwrapv -O3 -Wall -Wstrict-prototypes > > -I/var/folders/tm/b4drxjmn7j79gtp0ppbw5qlhgn/T/arrow-0.15.0.X.O0gHnUyz/apache-arrow-0.15.0/test-miniconda/envs/arrow-test/include > -arch x86_64 > -I/var/folders/tm/b4drxjmn7j79gtp0ppbw5qlhgn/T/arrow-0.15.0.X.O0gHnUyz/apache-arrow-0.15.0/test-miniconda/envs/arrow-test/include > -arch x86_64 > -I/var/folders/tm/b4drxjmn7j79gtp0ppbw5qlhgn/T/arrow-0.15.0.X.O0gHnUyz/apache-arrow-0.15.0/test-miniconda/envs/arrow-test/lib/python3.6/site-packages/numpy/core/include > > -I/private/var/folders/tm/b4drxjmn7j79gtp0ppbw5qlhgn/T/arrow-0.15.0.X.O0gHnUyz/apache-arrow-0.15.0/python/pyarrow/include > > -I/var/folders/tm/b4drxjmn7j79gtp0ppbw5qlhgn/T/arrow-0.15.0.X.O0gHnUyz/apache-arrow-0.15.0/test-miniconda/envs/arrow-test/include/python3.6m > -c pyarrow_cython_example.cpp -o > build/temp.macosx-10.7-x86_64-3.6/pyarrow_cython_example.o -std=c++11 > > pyarrow_cython_example.cpp:634:10: fatal error: 'unordered_map' file not > found > #include > ^~~ > 1 error generated. > error: command 'gcc' failed with exit status 1 > > > It also failed in all the tests in > pyarrow/tests/test_plasma.py::TestPlasmaClient: > > WARNING: Logging before InitGoogleLogging() is written to STDERR > I0927 08:54:13.048408 2902139840 store.cc:1149] Allowing the Plasma > store to use up to 0.1GB of memory. > I0927 08:54:13.049072 2902139840 store.cc:1176] Starting object store > with directory /tmp and huge page support disabled > E0927 08:54:13.049446 2902139840 io.cc:137] Socket pathname is too long. > F0927 08:54:13.049515 2902139840 store.cc:1074] Check failed: socket >= 0 > *** Check failure stack trace: *** > *** Aborted at 1569588853 (unix time) try "date -d @1569588853" if you > are using GNU date *** > PC: @0x0 (unknown) > *** SIGABRT (@0x7fffa41c7d42) received by PID 61396 (TID > 0x7fffacfb23c0) stack trace: *** > @ 0x7fffa42a8b3a _sigtramp > > For this latter issue, I believe we ran into this with Flight last > release. MacOS generates very long temporary directory paths, and this > usually exceeds the limit on domain socket path lengths. (I also can't > get mktemp to respect TMPDIR.) > > Best, > David > > > On 9/27/19, Micah Kornfield wrote: > > Hi, > > > > > > I would like to propose the following release candidate (RC1) of Apache > > > > Arrow version 0.15.0. This is a release consisting of 672 > > > > resolved JIRA issues[1]. (RC0 was aborted due to release script issues). > > > > > > This release candidate is based on commit: > > > > 77f091888dbe5a4e023a66b986a2dd474696061a [2] > > > > > > The source release rc1 is hosted at [3]. > > > > The binary artifacts are hosted at [4][5][6][7]. > > > > The changelog is located at [8]. > > > > > > Please download, verify checksums and signatures, run the unit tests, > > > > and vote on the release. See [9] for how to validate a release candidate. > > > > > > The vote will be open for at least 72 hours. > > > > > > [ ] +1 Release this as Apache Arrow 0.15.0 > > > > [ ] +0 > > > > [ ] -1 Do not release this as Apache Arrow 0.15.0 because... > > > > > > [1]: > > > https://issues.apache.org/jira/issues/?jql=project%20%3D%20ARROW%20AND%20status%20in%20%28Resolved%2C%20Closed%29%20AND%20fixVersion%20%3D%200.15.0 > > > > [2]: > > > https://github.com/apache/arrow/tree/77f091888dbe5a4e023a66b986a2dd474696061a > > > > [3]: > https://dist.apache.org/repos/dist/dev/arrow/apache-arrow-0.15.0-rc1 > > > > [4]: https://bintray.com/apache/arrow/centos-rc/0.15.0-rc1 >
Re: Travis CI delays
On Fri, Sep 27, 2019 at 3:25 PM Andy Grove wrote: > I've been poking around on the Arrow website and wiki and I can't find > documentation relating to CI. Do we have any documentation on how things > work today or what the goals are? I don't think so. > For Rust builds it isn't immediately > obvious why they are building on both Travis CI and Ursabot. > We have another thread [1], where we discuss multiple things about Buildbot (ursabot). I've enabled the buildbot builder for rust because it provides much quicker feedback than travis does. [1]: https://lists.apache.org/thread.html/02f7981176b67a12618b96b7d3b13e38b8f862e14c735dcf0ae359e0@%3Cdev.arrow.apache.org%3E > > > > On Fri, Sep 27, 2019 at 6:33 AM Wes McKinney wrote: > > > FYI, using the kibble.dev link from INFRA-18533, it seems that this > > September we're using about 15% of the ASF's total Travis CI capacity > > (60 concurrent workers I think) > > > > https://imgur.com/a/oOrbPsj > > > > The highest is Apache Druid (incubating) at 18%, so we are #2. Suffice > > to say the ASF's Travis couldn't accommodate us if we had twice as > > many pull requests > > > > On Fri, Sep 27, 2019 at 7:26 AM Wes McKinney > wrote: > > > > > > I've been sounding the alarm bells about this for a while. We need to > > > work to get ourselves off of Travis CI, but it is not going to be > > > easy.t > > > > > On Thu, Sep 26, 2019 at 11:42 PM Micah Kornfield < > emkornfi...@gmail.com> > > wrote: > > > > > > > > My understanding is the Travis CI queue is shared among all apache > > > > projects, and there are few including Arrow that make heavy use of > the > > > > resources. Hence, a lot of time waiting for jobs to start. I think > > there > > > > are some open JIRAs to finish dockerization of builds, I don't know > the > > > > current status of finding alternative CI sources though. > > > > > > > > On Thu, Sep 26, 2019 at 10:24 PM Andy Grove > > wrote: > > > > > > > > > I know this has been discussed in the past, and I apologize for not > > paying > > > > > attention at the time (and searching for arrow + travis in email > > isn't very > > > > > effective) but why does it take so long for our Travis CI builds > and > > are > > > > > there open JIRA issues related to this? > > > > > > > > > > Thanks, > > > > > > > > > > Andy. > > > > > > > >
Re: Travis CI delays
There are a bunch of other mailing list discussions over the last few months I recommend digging up and reviewing. As one practical issue with the Ursabot builds, those are running on servers that are hosted at my physical home in Nashville. If there's a power outage https://ci.ursalabs.org and all the CI builds will go down. If you're comfortable turning off the Rust Travis builds and relying on the Ursabot builds, and docker-compose for running the build locally, you're free to do that, but I think you need to make sure you have a contingency plan for verifying patches for the times when the network is unavailable. I travel about a third of the time and have no backup support if it goes down while I'm away from home. - Wes On Fri, Sep 27, 2019 at 8:25 AM Andy Grove wrote: > > I've been poking around on the Arrow website and wiki and I can't find > documentation relating to CI. Do we have any documentation on how things > work today or what the goals are? For Rust builds it isn't immediately > obvious why they are building on both Travis CI and Ursabot. > > > > On Fri, Sep 27, 2019 at 6:33 AM Wes McKinney wrote: > > > FYI, using the kibble.dev link from INFRA-18533, it seems that this > > September we're using about 15% of the ASF's total Travis CI capacity > > (60 concurrent workers I think) > > > > https://imgur.com/a/oOrbPsj > > > > The highest is Apache Druid (incubating) at 18%, so we are #2. Suffice > > to say the ASF's Travis couldn't accommodate us if we had twice as > > many pull requests > > > > On Fri, Sep 27, 2019 at 7:26 AM Wes McKinney wrote: > > > > > > I've been sounding the alarm bells about this for a while. We need to > > > work to get ourselves off of Travis CI, but it is not going to be > > > easy. > > > > > > On Thu, Sep 26, 2019 at 11:42 PM Micah Kornfield > > wrote: > > > > > > > > My understanding is the Travis CI queue is shared among all apache > > > > projects, and there are few including Arrow that make heavy use of the > > > > resources. Hence, a lot of time waiting for jobs to start. I think > > there > > > > are some open JIRAs to finish dockerization of builds, I don't know the > > > > current status of finding alternative CI sources though. > > > > > > > > On Thu, Sep 26, 2019 at 10:24 PM Andy Grove > > wrote: > > > > > > > > > I know this has been discussed in the past, and I apologize for not > > paying > > > > > attention at the time (and searching for arrow + travis in email > > isn't very > > > > > effective) but why does it take so long for our Travis CI builds and > > are > > > > > there open JIRA issues related to this? > > > > > > > > > > Thanks, > > > > > > > > > > Andy. > > > > > > >
Re: Subject: [VOTE] Release Apache Arrow 0.15.0 - RC1
I ran the source verification script on MacOS 10.12 (Sierra). I found that Micah's GPG key did not seem to be in KEYS and had to import it manually. Python failed in pyarrow/tests/test_cython.py::test_cython_api when invoking gcc. Is this the MacOS conda toolchain issue? It works if I add `-stdlib=libc++` to the compiler options (https://stackoverflow.com/questions/40301619) Compiling pyarrow_cython_example.pyx because it changed. [1/1] Cythonizing pyarrow_cython_example.pyx Extension module: ['/var/folders/tm/b4drxjmn7j79gtp0ppbw5qlhgn/T/arrow-0.15.0.X.O0gHnUyz/apache-arrow-0.15.0/test-miniconda/envs/arrow-test/lib/python3.6/site-packages/numpy/core/include', '/private/var/folders/tm/b4drxjmn7j79gtp0ppbw5qlhgn/T/arrow-0.15.0.X.O0gHnUyz/apache-arrow-0.15.0/python/pyarrow/include'] ['arrow', 'arrow_python'] ['/private/var/folders/tm/b4drxjmn7j79gtp0ppbw5qlhgn/T/arrow-0.15.0.X.O0gHnUyz/apache-arrow-0.15.0/python/pyarrow', '/var/folders/tm/b4drxjmn7j79gtp0ppbw5qlhgn/T/arrow-0.15.0.X.O0gHnUyz/install/lib'] running build_ext building 'pyarrow_cython_example' extension creating build creating build/temp.macosx-10.7-x86_64-3.6 gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/var/folders/tm/b4drxjmn7j79gtp0ppbw5qlhgn/T/arrow-0.15.0.X.O0gHnUyz/apache-arrow-0.15.0/test-miniconda/envs/arrow-test/include -arch x86_64 -I/var/folders/tm/b4drxjmn7j79gtp0ppbw5qlhgn/T/arrow-0.15.0.X.O0gHnUyz/apache-arrow-0.15.0/test-miniconda/envs/arrow-test/include -arch x86_64 -I/var/folders/tm/b4drxjmn7j79gtp0ppbw5qlhgn/T/arrow-0.15.0.X.O0gHnUyz/apache-arrow-0.15.0/test-miniconda/envs/arrow-test/lib/python3.6/site-packages/numpy/core/include -I/private/var/folders/tm/b4drxjmn7j79gtp0ppbw5qlhgn/T/arrow-0.15.0.X.O0gHnUyz/apache-arrow-0.15.0/python/pyarrow/include -I/var/folders/tm/b4drxjmn7j79gtp0ppbw5qlhgn/T/arrow-0.15.0.X.O0gHnUyz/apache-arrow-0.15.0/test-miniconda/envs/arrow-test/include/python3.6m -c pyarrow_cython_example.cpp -o build/temp.macosx-10.7-x86_64-3.6/pyarrow_cython_example.o -std=c++11 pyarrow_cython_example.cpp:634:10: fatal error: 'unordered_map' file not found #include ^~~ 1 error generated. error: command 'gcc' failed with exit status 1 It also failed in all the tests in pyarrow/tests/test_plasma.py::TestPlasmaClient: WARNING: Logging before InitGoogleLogging() is written to STDERR I0927 08:54:13.048408 2902139840 store.cc:1149] Allowing the Plasma store to use up to 0.1GB of memory. I0927 08:54:13.049072 2902139840 store.cc:1176] Starting object store with directory /tmp and huge page support disabled E0927 08:54:13.049446 2902139840 io.cc:137] Socket pathname is too long. F0927 08:54:13.049515 2902139840 store.cc:1074] Check failed: socket >= 0 *** Check failure stack trace: *** *** Aborted at 1569588853 (unix time) try "date -d @1569588853" if you are using GNU date *** PC: @0x0 (unknown) *** SIGABRT (@0x7fffa41c7d42) received by PID 61396 (TID 0x7fffacfb23c0) stack trace: *** @ 0x7fffa42a8b3a _sigtramp For this latter issue, I believe we ran into this with Flight last release. MacOS generates very long temporary directory paths, and this usually exceeds the limit on domain socket path lengths. (I also can't get mktemp to respect TMPDIR.) Best, David On 9/27/19, Micah Kornfield wrote: > Hi, > > > I would like to propose the following release candidate (RC1) of Apache > > Arrow version 0.15.0. This is a release consisting of 672 > > resolved JIRA issues[1]. (RC0 was aborted due to release script issues). > > > This release candidate is based on commit: > > 77f091888dbe5a4e023a66b986a2dd474696061a [2] > > > The source release rc1 is hosted at [3]. > > The binary artifacts are hosted at [4][5][6][7]. > > The changelog is located at [8]. > > > Please download, verify checksums and signatures, run the unit tests, > > and vote on the release. See [9] for how to validate a release candidate. > > > The vote will be open for at least 72 hours. > > > [ ] +1 Release this as Apache Arrow 0.15.0 > > [ ] +0 > > [ ] -1 Do not release this as Apache Arrow 0.15.0 because... > > > [1]: > https://issues.apache.org/jira/issues/?jql=project%20%3D%20ARROW%20AND%20status%20in%20%28Resolved%2C%20Closed%29%20AND%20fixVersion%20%3D%200.15.0 > > [2]: > https://github.com/apache/arrow/tree/77f091888dbe5a4e023a66b986a2dd474696061a > > [3]: https://dist.apache.org/repos/dist/dev/arrow/apache-arrow-0.15.0-rc1 > > [4]: https://bintray.com/apache/arrow/centos-rc/0.15.0-rc1 > > [5]: https://bintray.com/apache/arrow/debian-rc/0.15.0-rc1 > > [6]: https://bintray.com/apache/arrow/python-rc/0.15.0-rc1 > > [7]: https://bintray.com/apache/arrow/ubuntu-rc/0.15.0-rc1 > > [8]: > https://github.com/apache/arrow/blob/77f091888dbe5a4e023a66b986a2dd474696061a/CHANGELOG.md > > [9]: > https://cwiki.apache.org/confluence/display/ARROW/How+to+Verify+Release+Candidates >
Re: Travis CI delays
I've been poking around on the Arrow website and wiki and I can't find documentation relating to CI. Do we have any documentation on how things work today or what the goals are? For Rust builds it isn't immediately obvious why they are building on both Travis CI and Ursabot. On Fri, Sep 27, 2019 at 6:33 AM Wes McKinney wrote: > FYI, using the kibble.dev link from INFRA-18533, it seems that this > September we're using about 15% of the ASF's total Travis CI capacity > (60 concurrent workers I think) > > https://imgur.com/a/oOrbPsj > > The highest is Apache Druid (incubating) at 18%, so we are #2. Suffice > to say the ASF's Travis couldn't accommodate us if we had twice as > many pull requests > > On Fri, Sep 27, 2019 at 7:26 AM Wes McKinney wrote: > > > > I've been sounding the alarm bells about this for a while. We need to > > work to get ourselves off of Travis CI, but it is not going to be > > easy. > > > > On Thu, Sep 26, 2019 at 11:42 PM Micah Kornfield > wrote: > > > > > > My understanding is the Travis CI queue is shared among all apache > > > projects, and there are few including Arrow that make heavy use of the > > > resources. Hence, a lot of time waiting for jobs to start. I think > there > > > are some open JIRAs to finish dockerization of builds, I don't know the > > > current status of finding alternative CI sources though. > > > > > > On Thu, Sep 26, 2019 at 10:24 PM Andy Grove > wrote: > > > > > > > I know this has been discussed in the past, and I apologize for not > paying > > > > attention at the time (and searching for arrow + travis in email > isn't very > > > > effective) but why does it take so long for our Travis CI builds and > are > > > > there open JIRA issues related to this? > > > > > > > > Thanks, > > > > > > > > Andy. > > > > >
[jira] [Created] (ARROW-6724) [C++] Add simpler static ctor for BufferOutputStream than the current Create function
Wes McKinney created ARROW-6724:
---
Summary: [C++] Add simpler static ctor for BufferOutputStream than
the current Create function
Key: ARROW-6724
URL: https://issues.apache.org/jira/browse/ARROW-6724
Project: Apache Arrow
Issue Type: Improvement
Components: C++
Reporter: Wes McKinney
Not a major rough edge but the current {{Create}} function strikes me as a bit
awkward since a size and memory pool must be explicitly passed
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: Travis CI delays
I've been sounding the alarm bells about this for a while. We need to work to get ourselves off of Travis CI, but it is not going to be easy. On Thu, Sep 26, 2019 at 11:42 PM Micah Kornfield wrote: > > My understanding is the Travis CI queue is shared among all apache > projects, and there are few including Arrow that make heavy use of the > resources. Hence, a lot of time waiting for jobs to start. I think there > are some open JIRAs to finish dockerization of builds, I don't know the > current status of finding alternative CI sources though. > > On Thu, Sep 26, 2019 at 10:24 PM Andy Grove wrote: > > > I know this has been discussed in the past, and I apologize for not paying > > attention at the time (and searching for arrow + travis in email isn't very > > effective) but why does it take so long for our Travis CI builds and are > > there open JIRA issues related to this? > > > > Thanks, > > > > Andy. > >
Re: Travis CI delays
FYI, using the kibble.dev link from INFRA-18533, it seems that this September we're using about 15% of the ASF's total Travis CI capacity (60 concurrent workers I think) https://imgur.com/a/oOrbPsj The highest is Apache Druid (incubating) at 18%, so we are #2. Suffice to say the ASF's Travis couldn't accommodate us if we had twice as many pull requests On Fri, Sep 27, 2019 at 7:26 AM Wes McKinney wrote: > > I've been sounding the alarm bells about this for a while. We need to > work to get ourselves off of Travis CI, but it is not going to be > easy. > > On Thu, Sep 26, 2019 at 11:42 PM Micah Kornfield > wrote: > > > > My understanding is the Travis CI queue is shared among all apache > > projects, and there are few including Arrow that make heavy use of the > > resources. Hence, a lot of time waiting for jobs to start. I think there > > are some open JIRAs to finish dockerization of builds, I don't know the > > current status of finding alternative CI sources though. > > > > On Thu, Sep 26, 2019 at 10:24 PM Andy Grove wrote: > > > > > I know this has been discussed in the past, and I apologize for not paying > > > attention at the time (and searching for arrow + travis in email isn't > > > very > > > effective) but why does it take so long for our Travis CI builds and are > > > there open JIRA issues related to this? > > > > > > Thanks, > > > > > > Andy. > > >
[DISCUSS][Java] Reduce the range of synchronized block when releasing an ArrowBuf
Dear all,
When releasing an ArrowBuf, we will run the following piece of code:
private int decrement(int decrement) {
allocator.assertOpen();
final int outcome;
synchronized (allocationManager) {
outcome = bufRefCnt.addAndGet(-decrement);
if (outcome == 0) {
lDestructionTime = System.nanoTime();
allocationManager.release(this);
}
}
return outcome;
}
It can be seen that we need to acquire the lock for allocation manager
lock, no matter if we need to release the buffer. In addition, the
operation of decrementing refcount is only carried out after the lock is
acquired. This leads to unnecessary lock contention, and may degrade
performance.
We propose to change the code like this:
private int decrement(int decrement) {
allocator.assertOpen();
final int outcome;
outcome = bufRefCnt.addAndGet(-decrement);
if (outcome == 0) {
lDestructionTime = System.nanoTime();
synchronized (allocationManager) {
allocationManager.release(this);
}
}
return outcome;
}
Note that this change can be dangerous, as it lies in the core of our code
base, so we should be careful with it. On the other hand, it may have
non-trivial performance implication. For example, when a distributed task
is getting closed, a large number of ArrowBuf will be closed
simultaneously. If we reduce the range of the synchronization block, we can
significantly improve the performance.
Would you please give your valueable feedback?
Best,
Liya Fan
[jira] [Created] (ARROW-6723) [Java] Reduce the range of synchronized block when releasing an ArrowBuf
Liya Fan created ARROW-6723:
---
Summary: [Java] Reduce the range of synchronized block when
releasing an ArrowBuf
Key: ARROW-6723
URL: https://issues.apache.org/jira/browse/ARROW-6723
Project: Apache Arrow
Issue Type: Improvement
Components: Java
Reporter: Liya Fan
Assignee: Liya Fan
When releasing an ArrowBuf, we will run the following piece of code:
private int decrement(int decrement) {
allocator.assertOpen();
final int outcome;
synchronized (allocationManager) {
outcome = bufRefCnt.addAndGet(-decrement);
if (outcome == 0) {
lDestructionTime = System.nanoTime();
allocationManager.release(this);
}
}
return outcome;
}
It can be seen that we need to acquire the lock for allocation manager lock, no
matter if we need to release the buffer. In addition, the operation of
decrementing refcount is only carried out after the lock is acquired. This
leads to unnecessary resource contention, and may degrade performance.
We propose to change the code like this:
private int decrement(int decrement) {
allocator.assertOpen();
final int outcome;
outcome = bufRefCnt.addAndGet(-decrement);
if (outcome == 0) {
lDestructionTime = System.nanoTime();
synchronized (allocationManager) {
allocationManager.release(this);
}
}
return outcome;
}
Note that this change can be dangerous, as it lies in the core of our code
base, so we should be careful with it. On the other hand, it may have
non-trivial performance implication. As far as I know, when a distributed task
is getting closed, a large number of ArrowBuf will be closed simultaneously. If
we reduce the range of the synchronization block, we can significantly improve
the performance.
What do you think?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (ARROW-6722) [Java] Provide a uniform way to get vector name
Liya Fan created ARROW-6722: --- Summary: [Java] Provide a uniform way to get vector name Key: ARROW-6722 URL: https://issues.apache.org/jira/browse/ARROW-6722 Project: Apache Arrow Issue Type: Improvement Components: Java Reporter: Liya Fan Assignee: Liya Fan Currently, the getName method is defined in BaseValueVector, as an abstract class. However, some vector does not extend the BaseValueVector, like StructVector, UnionVector, ZeroVector. In this issue, we move the method to ValueVector interface, the base interface for all vectors. This makes it easier to get a vector's name without checking its type. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ARROW-6721) [JAVA] Avro adapter benchmark only runs once in JMH
Ji Liu created ARROW-6721:
-
Summary: [JAVA] Avro adapter benchmark only runs once in JMH
Key: ARROW-6721
URL: https://issues.apache.org/jira/browse/ARROW-6721
Project: Apache Arrow
Issue Type: Bug
Components: Java
Reporter: Ji Liu
Assignee: Ji Liu
The current {{AvroAdapterBenchmark}} actually only run once during JMH
evaluation, since the decoder was consumed for the first time and the follow-up
invokes will directly return.
To solve this, we use {{BinaryDecoder}} explicitly in benchmark and reset its
inner stream first when the test method is invoked.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Subject: [VOTE] Release Apache Arrow 0.15.0 - RC1
Hi, I would like to propose the following release candidate (RC1) of Apache Arrow version 0.15.0. This is a release consisting of 672 resolved JIRA issues[1]. (RC0 was aborted due to release script issues). This release candidate is based on commit: 77f091888dbe5a4e023a66b986a2dd474696061a [2] The source release rc1 is hosted at [3]. The binary artifacts are hosted at [4][5][6][7]. The changelog is located at [8]. Please download, verify checksums and signatures, run the unit tests, and vote on the release. See [9] for how to validate a release candidate. The vote will be open for at least 72 hours. [ ] +1 Release this as Apache Arrow 0.15.0 [ ] +0 [ ] -1 Do not release this as Apache Arrow 0.15.0 because... [1]: https://issues.apache.org/jira/issues/?jql=project%20%3D%20ARROW%20AND%20status%20in%20%28Resolved%2C%20Closed%29%20AND%20fixVersion%20%3D%200.15.0 [2]: https://github.com/apache/arrow/tree/77f091888dbe5a4e023a66b986a2dd474696061a [3]: https://dist.apache.org/repos/dist/dev/arrow/apache-arrow-0.15.0-rc1 [4]: https://bintray.com/apache/arrow/centos-rc/0.15.0-rc1 [5]: https://bintray.com/apache/arrow/debian-rc/0.15.0-rc1 [6]: https://bintray.com/apache/arrow/python-rc/0.15.0-rc1 [7]: https://bintray.com/apache/arrow/ubuntu-rc/0.15.0-rc1 [8]: https://github.com/apache/arrow/blob/77f091888dbe5a4e023a66b986a2dd474696061a/CHANGELOG.md [9]: https://cwiki.apache.org/confluence/display/ARROW/How+to+Verify+Release+Candidates
