[jira] [Created] (ARROW-7731) [Parquet] Support LargeListArray
marc abboud created ARROW-7731:
--
Summary: [Parquet] Support LargeListArray
Key: ARROW-7731
URL: https://issues.apache.org/jira/browse/ARROW-7731
Project: Apache Arrow
Issue Type: Improvement
Reporter: marc abboud
For now it's not possible to write a pyarrow.Table containing a LargeListArray
in parquet. The lines
{code:java}
from pyarrow import parquet
import pyarrow as pa
indices = [1, 2, 3]
indptr = [0, 1, 2, 3]
q = pa.lib.LargeListArray.from_arrays(indptr, indices)
table = pa.Table.from_arrays([q], names=['no'])
parquet.write_table(table, '/test'){code}
yields the error
{code:java}
ArrowNotImplementedError: Unhandled type for Arrow to Parquet schema
conversion: large_list
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (ARROW-7732) [Python][C++] Parquet statistics wrong for pandas Categorical
Florian Jetter created ARROW-7732:
-
Summary: [Python][C++] Parquet statistics wrong for pandas
Categorical
Key: ARROW-7732
URL: https://issues.apache.org/jira/browse/ARROW-7732
Project: Apache Arrow
Issue Type: Bug
Components: Python
Affects Versions: 0.15.1, 0.16.0
Reporter: Florian Jetter
h3. Observed behaviour
Statistics for categorical data are equivalent for all row groups and refer to
the entire {{CategoricalDtype}} instead of the data included in the row group.
h3. Expected behaviour
The row group statistics should only include data which is part of the actual
row group, not the entire {{CategoricalDtype}}
h3. Minimal example
{code:python}
import pandas as pd
import pyarrow as pa
import pyarrow.parquet as pq
test_df = pd.DataFrame({"categorical": pd.Categorical(["1", "42"])})
table = pa.Table.from_pandas(test_df)
pq.write_table(
table,
"test_parquet",
chunk_size=1,
)
test_parquet = pq.ParquetFile("test_parquet")
test_parquet.metadata.row_group(0).column(0).statistics
{code}
{code:java}
Out[1]:
has_min_max: True
min: 1
max: 42
null_count: 0
distinct_count: 0
num_values: 1
physical_type: BYTE_ARRAY
logical_type: String
converted_type (legacy): UTF8
{code}
Expected would be
{{min:1}} {{max:1}} instead of {{max: 42}} for the first row group
Tested with
pandas==1.0.0
pyarrow==bd08d0ecbe355b9e0de7d07e8b9ff6ccdb150e73 (current master /
essentially 0.16.0)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: [VOTE] Release Apache Arrow 0.16.0 - RC1
On Thu, Jan 30, 2020 at 10:24 PM Sutou Kouhei wrote: > > Hi, > > Sorry for my late summary. > > I verified RC1 and have been fixed some problems. > Here are remained problems I found: > > * "yum install"/"dnf install" don't work. It seems that > Yum repository's metadata is broken. It may be a our new > binary upload script problem. The script may have a > retry related problem. Please verify that it works with RC2, I had to restart the upload script a couple of times. > > * "dev/release/verify-release-candidate.sh wheels 0.16.0 1" > doesn't work. See the following log for details. I > think that this is a verify script problem but I don't > know how to fix it. It may be a Conda related problem. > > https://github.com/apache/arrow/blob/master/dev/release/verify-release-candidate.sh#L613 > > > + conda install -y --file > /home/kou/work/cpp/arrow.kou/ci/conda_env_python.yml pandas > + '[' 5 -lt 1 ']' > + local cmd=install > + shift > + case "$cmd" in > + > OLDPATH=/tmp/arrow-0.16.0.j9Uct/test-miniconda/envs/_verify_wheel-2.7mu/bin:/tmp/arrow-0.16.0.j9Uct/test-miniconda/condabin:/home/kou/work/go/bin:/bin:/home/kou/local/bin:/home/kou/.config/composer/vendor/bin:/var/lib/gems/2.5.0/bin:/usr/local/bin:/usr/bin:/usr/games > + __add_sys_prefix_to_path > + '[' -n '' ']' > ++ dirname /tmp/arrow-0.16.0.j9Uct/test-miniconda/bin/conda > + SYSP=/tmp/arrow-0.16.0.j9Uct/test-miniconda/bin > ++ dirname /tmp/arrow-0.16.0.j9Uct/test-miniconda/bin > + SYSP=/tmp/arrow-0.16.0.j9Uct/test-miniconda > + '[' -n '' ']' > + > PATH=/tmp/arrow-0.16.0.j9Uct/test-miniconda/bin:/tmp/arrow-0.16.0.j9Uct/test-miniconda/envs/_verify_wheel-2.7mu/bin:/tmp/arrow-0.16.0.j9Uct/test-miniconda/condabin:/home/kou/work/go/bin:/bin:/home/kou/local/bin:/home/kou/.config/composer/vendor/bin:/var/lib/gems/2.5.0/bin:/usr/local/bin:/usr/bin:/usr/games > + export PATH > + /tmp/arrow-0.16.0.j9Uct/test-miniconda/bin/conda install -y --file > /home/kou/work/cpp/arrow.kou/ci/conda_env_python.yml pandas > Collecting package metadata (current_repodata.json): ...working... done > Solving environment: ...working... failed with initial frozen solve. Retrying > with flexible solve. > Collecting package metadata (repodata.json): ...working... done > Solving environment: ...working... failed with initial frozen solve. Retrying > with flexible solve. > > PackagesNotFoundError: The following packages are not available from current > channels: We need to use `--channel conda-forge` in all of the conda commands. Creating a PR. Thanks, Krisztian > > - pytest-faulthandler > - pytest-lazy-fixture > > Current channels: > > - https://repo.anaconda.com/pkgs/main/linux-64 > - https://repo.anaconda.com/pkgs/main/noarch > - https://repo.anaconda.com/pkgs/r/linux-64 > - https://repo.anaconda.com/pkgs/r/noarch > > To search for alternate channels that may provide the conda package you're > looking for, navigate to > > https://anaconda.org > > and use the search bar at the top of the page. > > > Thanks, > -- > kou > > In > "Re: [VOTE] Release Apache Arrow 0.16.0 - RC1" on Thu, 30 Jan 2020 21:52:39 > +0100, > Krisztián Szűcs wrote: > > > Hi, > > > > RC2 is in progress. The source is uploaded, I'm waiting for the binaries. > > Thanks everyone for the help! > > > > - Krisztian > > > > On Thu, Jan 30, 2020 at 12:50 AM Krisztián Szűcs > > wrote: > >> > >> Let's try to fix it, then we can cut RC2 afterwards > >> > >> Thanks for your help! > >> > >> On Wed, Jan 29, 2020 at 10:46 PM Wes McKinney wrote: > >> > > >> > I just commented on the issue. Seems likely to be fallout from > >> > ARROW-3789 and so I think we should fix it > >> > > >> > On Wed, Jan 29, 2020 at 3:19 PM Bryan Cutler wrote: > >> > > > >> > > An update on Spark integration tests: the new error looks to be a > >> > > regression so I made https://issues.apache.org/jira/browse/ARROW-7723 > >> > > and > >> > > marked as a blocker. It's possible to work around this bug, so I > >> > > wouldn't > >> > > call it a hard blocker if we need to proceed with the release. > >> > > > >> > > On Wed, Jan 29, 2020 at 7:45 AM Neal Richardson > >> > > > >> > > wrote: > >> > > > >> > > > The place where the segfault is triggered in the R nightlies is a > >> > > > couple of > >> > > > tests after the one I added in that patch. If that patch is causing > >> > > > the > >> > > > segfaults, we can skip the new test ( > >> > > > > >> > > > https://github.com/apache/arrow/blob/master/r/tests/testthat/test-parquet.R#L125 > >> > > > ) > >> > > > and investigate later. The patch is exercising previously existing > >> > > > codepaths that were not tested, so I don't think that identifying and > >> > > > fixing the segfault should be release blocking (though we should > >> > > > clearly > >> > > > fix it). > >> > > > > >> > > > Neal > >> > > > > >> > > > > >> > > > > >> > > > On Wed, Jan 29, 2020 at 7:33 AM David Li > >> > > > wrote: > >> > > > > >> > > > > The Flight leak sh
Re: [VOTE] Release Apache Arrow 0.16.0 - RC2
Hi, "dev/release/verify-release-candidate.sh binaries 0.16.0 2" is still failing. And I found the cause of the problem. Could you re-run 03-binary.sh with the change? --- diff --git a/dev/release/binary-task.rb b/dev/release/binary-task.rb index 4b5960445..3dd1d9c08 100644 --- a/dev/release/binary-task.rb +++ b/dev/release/binary-task.rb @@ -1015,6 +1015,7 @@ class BinaryTask ["debian", "buster", "main"], ["ubuntu", "xenial", "main"], ["ubuntu", "bionic", "main"], + ["ubuntu", "cosmic", "main"], ["ubuntu", "disco", "main"], ["ubuntu", "eoan", "main"], ] --- Existing APT metadata files for Ubuntu Cosmic has problem (.sha512 file is missing). So we need to re-generate it only for 0.16.0 release. We don't need to do this for the next release. Thanks, -- kou In "[VOTE] Release Apache Arrow 0.16.0 - RC2" on Fri, 31 Jan 2020 04:13:12 +0100, Krisztián Szűcs wrote: > Hi, > > I would like to propose the following release candidate (RC2) of Apache > Arrow version 0.16.0. This is a release consisting of 728 > resolved JIRA issues[1]. > > This release candidate is based on commit: > 729a7689fd87572e6a14ad36f19cd579a8b8d9c5 [2] > > The source release rc2 is hosted at [3]. > The binary artifacts are hosted at [4][5][6][7]. > The changelog is located at [8]. > > Please download, verify checksums and signatures, run the unit tests, > and vote on the release. See [9] for how to validate a release candidate. > > The vote will be open for at least 72 hours. > > [ ] +1 Release this as Apache Arrow 0.16.0 > [ ] +0 > [ ] -1 Do not release this as Apache Arrow 0.16.0 because... > > [1]: > https://issues.apache.org/jira/issues/?jql=project%20%3D%20ARROW%20AND%20status%20in%20%28Resolved%2C%20Closed%29%20AND%20fixVersion%20%3D%200.16.0 > [2]: > https://github.com/apache/arrow/tree/729a7689fd87572e6a14ad36f19cd579a8b8d9c5 > [3]: https://dist.apache.org/repos/dist/dev/arrow/apache-arrow-0.16.0-rc2 > [4]: https://bintray.com/apache/arrow/centos-rc/0.16.0-rc2 > [5]: https://bintray.com/apache/arrow/debian-rc/0.16.0-rc2 > [6]: https://bintray.com/apache/arrow/python-rc/0.16.0-rc2 > [7]: https://bintray.com/apache/arrow/ubuntu-rc/0.16.0-rc2 > [8]: > https://github.com/apache/arrow/blob/729a7689fd87572e6a14ad36f19cd579a8b8d9c5/CHANGELOG.md > [9]: > https://cwiki.apache.org/confluence/display/ARROW/How+to+Verify+Release+Candidates
Re: [VOTE] Release Apache Arrow 0.16.0 - RC2
Hi, One more thing. It seems that some .deb/.rpm files for arm64 aren't uploaded. Could you re-run "crossbow download-artifacts build-730" and re-run "dev/release/03-binary.sh"? Thanks, -- kou In <20200131.205531.1001999778589734260@clear-code.com> "Re: [VOTE] Release Apache Arrow 0.16.0 - RC2" on Fri, 31 Jan 2020 20:55:31 +0900 (JST), Sutou Kouhei wrote: > Hi, > > "dev/release/verify-release-candidate.sh binaries 0.16.0 2" > is still failing. And I found the cause of the problem. > > Could you re-run 03-binary.sh with the change? > > --- > diff --git a/dev/release/binary-task.rb b/dev/release/binary-task.rb > index 4b5960445..3dd1d9c08 100644 > --- a/dev/release/binary-task.rb > +++ b/dev/release/binary-task.rb > @@ -1015,6 +1015,7 @@ class BinaryTask >["debian", "buster", "main"], >["ubuntu", "xenial", "main"], >["ubuntu", "bionic", "main"], > + ["ubuntu", "cosmic", "main"], >["ubuntu", "disco", "main"], >["ubuntu", "eoan", "main"], > ] > --- > > Existing APT metadata files for Ubuntu Cosmic has > problem (.sha512 file is missing). So we need to re-generate > it only for 0.16.0 release. We don't need to do this for the > next release. > > > Thanks, > -- > kou > > In > "[VOTE] Release Apache Arrow 0.16.0 - RC2" on Fri, 31 Jan 2020 04:13:12 > +0100, > Krisztián Szűcs wrote: > >> Hi, >> >> I would like to propose the following release candidate (RC2) of Apache >> Arrow version 0.16.0. This is a release consisting of 728 >> resolved JIRA issues[1]. >> >> This release candidate is based on commit: >> 729a7689fd87572e6a14ad36f19cd579a8b8d9c5 [2] >> >> The source release rc2 is hosted at [3]. >> The binary artifacts are hosted at [4][5][6][7]. >> The changelog is located at [8]. >> >> Please download, verify checksums and signatures, run the unit tests, >> and vote on the release. See [9] for how to validate a release candidate. >> >> The vote will be open for at least 72 hours. >> >> [ ] +1 Release this as Apache Arrow 0.16.0 >> [ ] +0 >> [ ] -1 Do not release this as Apache Arrow 0.16.0 because... >> >> [1]: >> https://issues.apache.org/jira/issues/?jql=project%20%3D%20ARROW%20AND%20status%20in%20%28Resolved%2C%20Closed%29%20AND%20fixVersion%20%3D%200.16.0 >> [2]: >> https://github.com/apache/arrow/tree/729a7689fd87572e6a14ad36f19cd579a8b8d9c5 >> [3]: https://dist.apache.org/repos/dist/dev/arrow/apache-arrow-0.16.0-rc2 >> [4]: https://bintray.com/apache/arrow/centos-rc/0.16.0-rc2 >> [5]: https://bintray.com/apache/arrow/debian-rc/0.16.0-rc2 >> [6]: https://bintray.com/apache/arrow/python-rc/0.16.0-rc2 >> [7]: https://bintray.com/apache/arrow/ubuntu-rc/0.16.0-rc2 >> [8]: >> https://github.com/apache/arrow/blob/729a7689fd87572e6a14ad36f19cd579a8b8d9c5/CHANGELOG.md >> [9]: >> https://cwiki.apache.org/confluence/display/ARROW/How+to+Verify+Release+Candidates
Re: [VOTE] Release Apache Arrow 0.16.0 - RC2
On Fri, Jan 31, 2020 at 12:57 PM Sutou Kouhei wrote: > > Hi, > > One more thing. > > It seems that some .deb/.rpm files for arm64 aren't > uploaded. Could you re-run "crossbow download-artifacts > build-730" and re-run "dev/release/03-binary.sh"? Most of the ARM builds have exceeded the CI timeout [1], although I'll update the xenial, bionic, stretch packages. [1]: https://github.com/ursa-labs/crossbow/branches/all?query=arm > > > Thanks, > -- > kou > > In <20200131.205531.1001999778589734260@clear-code.com> > "Re: [VOTE] Release Apache Arrow 0.16.0 - RC2" on Fri, 31 Jan 2020 20:55:31 > +0900 (JST), > Sutou Kouhei wrote: > > > Hi, > > > > "dev/release/verify-release-candidate.sh binaries 0.16.0 2" > > is still failing. And I found the cause of the problem. > > > > Could you re-run 03-binary.sh with the change? > > > > --- > > diff --git a/dev/release/binary-task.rb b/dev/release/binary-task.rb > > index 4b5960445..3dd1d9c08 100644 > > --- a/dev/release/binary-task.rb > > +++ b/dev/release/binary-task.rb > > @@ -1015,6 +1015,7 @@ class BinaryTask > >["debian", "buster", "main"], > >["ubuntu", "xenial", "main"], > >["ubuntu", "bionic", "main"], > > + ["ubuntu", "cosmic", "main"], > >["ubuntu", "disco", "main"], > >["ubuntu", "eoan", "main"], > > ] > > --- > > > > Existing APT metadata files for Ubuntu Cosmic has > > problem (.sha512 file is missing). So we need to re-generate > > it only for 0.16.0 release. We don't need to do this for the > > next release. > > > > > > Thanks, > > -- > > kou > > > > In > > "[VOTE] Release Apache Arrow 0.16.0 - RC2" on Fri, 31 Jan 2020 04:13:12 > > +0100, > > Krisztián Szűcs wrote: > > > >> Hi, > >> > >> I would like to propose the following release candidate (RC2) of Apache > >> Arrow version 0.16.0. This is a release consisting of 728 > >> resolved JIRA issues[1]. > >> > >> This release candidate is based on commit: > >> 729a7689fd87572e6a14ad36f19cd579a8b8d9c5 [2] > >> > >> The source release rc2 is hosted at [3]. > >> The binary artifacts are hosted at [4][5][6][7]. > >> The changelog is located at [8]. > >> > >> Please download, verify checksums and signatures, run the unit tests, > >> and vote on the release. See [9] for how to validate a release candidate. > >> > >> The vote will be open for at least 72 hours. > >> > >> [ ] +1 Release this as Apache Arrow 0.16.0 > >> [ ] +0 > >> [ ] -1 Do not release this as Apache Arrow 0.16.0 because... > >> > >> [1]: > >> https://issues.apache.org/jira/issues/?jql=project%20%3D%20ARROW%20AND%20status%20in%20%28Resolved%2C%20Closed%29%20AND%20fixVersion%20%3D%200.16.0 > >> [2]: > >> https://github.com/apache/arrow/tree/729a7689fd87572e6a14ad36f19cd579a8b8d9c5 > >> [3]: https://dist.apache.org/repos/dist/dev/arrow/apache-arrow-0.16.0-rc2 > >> [4]: https://bintray.com/apache/arrow/centos-rc/0.16.0-rc2 > >> [5]: https://bintray.com/apache/arrow/debian-rc/0.16.0-rc2 > >> [6]: https://bintray.com/apache/arrow/python-rc/0.16.0-rc2 > >> [7]: https://bintray.com/apache/arrow/ubuntu-rc/0.16.0-rc2 > >> [8]: > >> https://github.com/apache/arrow/blob/729a7689fd87572e6a14ad36f19cd579a8b8d9c5/CHANGELOG.md > >> [9]: > >> https://cwiki.apache.org/confluence/display/ARROW/How+to+Verify+Release+Candidates
Re: [VOTE] Release Apache Arrow 0.16.0 - RC2
On Ubuntu 18.04, the source verification is successful until the go step, which fails: https://gist.github.com/pitrou/7e089ac146197b1141585c271cb39866 Side note: the JS verification step should avoid spamming the terminal with tons of useless information. Regards Antoine. Le 31/01/2020 à 04:13, Krisztián Szűcs a écrit : > Hi, > > I would like to propose the following release candidate (RC2) of Apache > Arrow version 0.16.0. This is a release consisting of 728 > resolved JIRA issues[1]. > > This release candidate is based on commit: > 729a7689fd87572e6a14ad36f19cd579a8b8d9c5 [2] > > The source release rc2 is hosted at [3]. > The binary artifacts are hosted at [4][5][6][7]. > The changelog is located at [8]. > > Please download, verify checksums and signatures, run the unit tests, > and vote on the release. See [9] for how to validate a release candidate. > > The vote will be open for at least 72 hours.
[NIGHTLY] Arrow Build Report for Job nightly-2020-01-31-0
Arrow Build Report for Job nightly-2020-01-31-0 All tasks: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-01-31-0 Failed Tasks: - gandiva-jar-osx: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-01-31-0-travis-gandiva-jar-osx - wheel-manylinux2014-cp36m: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-01-31-0-azure-wheel-manylinux2014-cp36m Succeeded Tasks: - centos-6: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-01-31-0-azure-centos-6 - centos-7: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-01-31-0-azure-centos-7 - centos-8: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-01-31-0-azure-centos-8 - conda-linux-gcc-py27: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-01-31-0-azure-conda-linux-gcc-py27 - conda-linux-gcc-py36: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-01-31-0-azure-conda-linux-gcc-py36 - conda-linux-gcc-py37: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-01-31-0-azure-conda-linux-gcc-py37 - conda-linux-gcc-py38: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-01-31-0-azure-conda-linux-gcc-py38 - conda-osx-clang-py27: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-01-31-0-azure-conda-osx-clang-py27 - conda-osx-clang-py36: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-01-31-0-azure-conda-osx-clang-py36 - conda-osx-clang-py37: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-01-31-0-azure-conda-osx-clang-py37 - conda-osx-clang-py38: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-01-31-0-azure-conda-osx-clang-py38 - conda-win-vs2015-py36: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-01-31-0-azure-conda-win-vs2015-py36 - conda-win-vs2015-py37: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-01-31-0-azure-conda-win-vs2015-py37 - conda-win-vs2015-py38: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-01-31-0-azure-conda-win-vs2015-py38 - debian-buster: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-01-31-0-azure-debian-buster - debian-stretch: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-01-31-0-azure-debian-stretch - gandiva-jar-trusty: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-01-31-0-travis-gandiva-jar-trusty - homebrew-cpp: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-01-31-0-travis-homebrew-cpp - macos-r-autobrew: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-01-31-0-travis-macos-r-autobrew - test-conda-cpp: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-01-31-0-circle-test-conda-cpp - test-conda-python-2.7-pandas-latest: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-01-31-0-circle-test-conda-python-2.7-pandas-latest - test-conda-python-2.7: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-01-31-0-circle-test-conda-python-2.7 - test-conda-python-3.6: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-01-31-0-circle-test-conda-python-3.6 - test-conda-python-3.7-dask-latest: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-01-31-0-circle-test-conda-python-3.7-dask-latest - test-conda-python-3.7-hdfs-2.9.2: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-01-31-0-circle-test-conda-python-3.7-hdfs-2.9.2 - test-conda-python-3.7-pandas-latest: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-01-31-0-circle-test-conda-python-3.7-pandas-latest - test-conda-python-3.7-pandas-master: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-01-31-0-circle-test-conda-python-3.7-pandas-master - test-conda-python-3.7-spark-master: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-01-31-0-circle-test-conda-python-3.7-spark-master - test-conda-python-3.7-turbodbc-latest: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-01-31-0-circle-test-conda-python-3.7-turbodbc-latest - test-conda-python-3.7-turbodbc-master: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-01-31-0-circle-test-conda-python-3.7-turbodbc-master - test-conda-python-3.7: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-01-31-0-circle-test-conda-python-3.7 - test-conda-python-3.8-dask-master: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-01-31-0-circle-test-conda-python-3.8-dask-master - test-conda-p
Re: [VOTE] Release Apache Arrow 0.16.0 - RC2
Rerun with cosmic and the successfully built ARM packages. On Fri, Jan 31, 2020 at 12:57 PM Sutou Kouhei wrote: > > Hi, > > One more thing. > > It seems that some .deb/.rpm files for arm64 aren't > uploaded. Could you re-run "crossbow download-artifacts > build-730" and re-run "dev/release/03-binary.sh"? > > > Thanks, > -- > kou > > In <20200131.205531.1001999778589734260@clear-code.com> > "Re: [VOTE] Release Apache Arrow 0.16.0 - RC2" on Fri, 31 Jan 2020 20:55:31 > +0900 (JST), > Sutou Kouhei wrote: > > > Hi, > > > > "dev/release/verify-release-candidate.sh binaries 0.16.0 2" > > is still failing. And I found the cause of the problem. > > > > Could you re-run 03-binary.sh with the change? > > > > --- > > diff --git a/dev/release/binary-task.rb b/dev/release/binary-task.rb > > index 4b5960445..3dd1d9c08 100644 > > --- a/dev/release/binary-task.rb > > +++ b/dev/release/binary-task.rb > > @@ -1015,6 +1015,7 @@ class BinaryTask > >["debian", "buster", "main"], > >["ubuntu", "xenial", "main"], > >["ubuntu", "bionic", "main"], > > + ["ubuntu", "cosmic", "main"], > >["ubuntu", "disco", "main"], > >["ubuntu", "eoan", "main"], > > ] > > --- > > > > Existing APT metadata files for Ubuntu Cosmic has > > problem (.sha512 file is missing). So we need to re-generate > > it only for 0.16.0 release. We don't need to do this for the > > next release. > > > > > > Thanks, > > -- > > kou > > > > In > > "[VOTE] Release Apache Arrow 0.16.0 - RC2" on Fri, 31 Jan 2020 04:13:12 > > +0100, > > Krisztián Szűcs wrote: > > > >> Hi, > >> > >> I would like to propose the following release candidate (RC2) of Apache > >> Arrow version 0.16.0. This is a release consisting of 728 > >> resolved JIRA issues[1]. > >> > >> This release candidate is based on commit: > >> 729a7689fd87572e6a14ad36f19cd579a8b8d9c5 [2] > >> > >> The source release rc2 is hosted at [3]. > >> The binary artifacts are hosted at [4][5][6][7]. > >> The changelog is located at [8]. > >> > >> Please download, verify checksums and signatures, run the unit tests, > >> and vote on the release. See [9] for how to validate a release candidate. > >> > >> The vote will be open for at least 72 hours. > >> > >> [ ] +1 Release this as Apache Arrow 0.16.0 > >> [ ] +0 > >> [ ] -1 Do not release this as Apache Arrow 0.16.0 because... > >> > >> [1]: > >> https://issues.apache.org/jira/issues/?jql=project%20%3D%20ARROW%20AND%20status%20in%20%28Resolved%2C%20Closed%29%20AND%20fixVersion%20%3D%200.16.0 > >> [2]: > >> https://github.com/apache/arrow/tree/729a7689fd87572e6a14ad36f19cd579a8b8d9c5 > >> [3]: https://dist.apache.org/repos/dist/dev/arrow/apache-arrow-0.16.0-rc2 > >> [4]: https://bintray.com/apache/arrow/centos-rc/0.16.0-rc2 > >> [5]: https://bintray.com/apache/arrow/debian-rc/0.16.0-rc2 > >> [6]: https://bintray.com/apache/arrow/python-rc/0.16.0-rc2 > >> [7]: https://bintray.com/apache/arrow/ubuntu-rc/0.16.0-rc2 > >> [8]: > >> https://github.com/apache/arrow/blob/729a7689fd87572e6a14ad36f19cd579a8b8d9c5/CHANGELOG.md > >> [9]: > >> https://cwiki.apache.org/confluence/display/ARROW/How+to+Verify+Release+Candidates
Re: [NIGHTLY] Arrow Build Report for Job nightly-2020-01-31-0
Can we disable gandiva-jar-osx until there is a patch re-enabling and fixing it? On Fri, Jan 31, 2020 at 7:31 AM Crossbow wrote: > > > Arrow Build Report for Job nightly-2020-01-31-0 > > All tasks: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-01-31-0 > > Failed Tasks: > - gandiva-jar-osx: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-01-31-0-travis-gandiva-jar-osx > - wheel-manylinux2014-cp36m: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-01-31-0-azure-wheel-manylinux2014-cp36m > > Succeeded Tasks: > - centos-6: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-01-31-0-azure-centos-6 > - centos-7: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-01-31-0-azure-centos-7 > - centos-8: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-01-31-0-azure-centos-8 > - conda-linux-gcc-py27: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-01-31-0-azure-conda-linux-gcc-py27 > - conda-linux-gcc-py36: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-01-31-0-azure-conda-linux-gcc-py36 > - conda-linux-gcc-py37: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-01-31-0-azure-conda-linux-gcc-py37 > - conda-linux-gcc-py38: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-01-31-0-azure-conda-linux-gcc-py38 > - conda-osx-clang-py27: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-01-31-0-azure-conda-osx-clang-py27 > - conda-osx-clang-py36: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-01-31-0-azure-conda-osx-clang-py36 > - conda-osx-clang-py37: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-01-31-0-azure-conda-osx-clang-py37 > - conda-osx-clang-py38: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-01-31-0-azure-conda-osx-clang-py38 > - conda-win-vs2015-py36: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-01-31-0-azure-conda-win-vs2015-py36 > - conda-win-vs2015-py37: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-01-31-0-azure-conda-win-vs2015-py37 > - conda-win-vs2015-py38: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-01-31-0-azure-conda-win-vs2015-py38 > - debian-buster: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-01-31-0-azure-debian-buster > - debian-stretch: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-01-31-0-azure-debian-stretch > - gandiva-jar-trusty: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-01-31-0-travis-gandiva-jar-trusty > - homebrew-cpp: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-01-31-0-travis-homebrew-cpp > - macos-r-autobrew: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-01-31-0-travis-macos-r-autobrew > - test-conda-cpp: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-01-31-0-circle-test-conda-cpp > - test-conda-python-2.7-pandas-latest: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-01-31-0-circle-test-conda-python-2.7-pandas-latest > - test-conda-python-2.7: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-01-31-0-circle-test-conda-python-2.7 > - test-conda-python-3.6: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-01-31-0-circle-test-conda-python-3.6 > - test-conda-python-3.7-dask-latest: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-01-31-0-circle-test-conda-python-3.7-dask-latest > - test-conda-python-3.7-hdfs-2.9.2: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-01-31-0-circle-test-conda-python-3.7-hdfs-2.9.2 > - test-conda-python-3.7-pandas-latest: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-01-31-0-circle-test-conda-python-3.7-pandas-latest > - test-conda-python-3.7-pandas-master: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-01-31-0-circle-test-conda-python-3.7-pandas-master > - test-conda-python-3.7-spark-master: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-01-31-0-circle-test-conda-python-3.7-spark-master > - test-conda-python-3.7-turbodbc-latest: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-01-31-0-circle-test-conda-python-3.7-turbodbc-latest > - test-conda-python-3.7-turbodbc-master: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-01-31-0-circle-test-conda-python-
Re: [VOTE] Release Apache Arrow 0.16.0 - RC2
hi Antoine -- which version of Go are you using? Some of these issues are resolved by upgrading the Go version On Fri, Jan 31, 2020 at 6:19 AM Antoine Pitrou wrote: > > > On Ubuntu 18.04, the source verification is successful until the go > step, which fails: > https://gist.github.com/pitrou/7e089ac146197b1141585c271cb39866 > > Side note: the JS verification step should avoid spamming the terminal > with tons of useless information. > > Regards > > Antoine. > > > Le 31/01/2020 à 04:13, Krisztián Szűcs a écrit : > > Hi, > > > > I would like to propose the following release candidate (RC2) of Apache > > Arrow version 0.16.0. This is a release consisting of 728 > > resolved JIRA issues[1]. > > > > This release candidate is based on commit: > > 729a7689fd87572e6a14ad36f19cd579a8b8d9c5 [2] > > > > The source release rc2 is hosted at [3]. > > The binary artifacts are hosted at [4][5][6][7]. > > The changelog is located at [8]. > > > > Please download, verify checksums and signatures, run the unit tests, > > and vote on the release. See [9] for how to validate a release candidate. > > > > The vote will be open for at least 72 hours.
Re: [VOTE] Release Apache Arrow 0.16.0 - RC2
It's the version packaged in Ubuntu 18.04. $ go version go version go1.10.4 linux/amd64 Regards Antoine. Le 31/01/2020 à 20:06, Wes McKinney a écrit : > hi Antoine -- which version of Go are you using? Some of these issues > are resolved by upgrading the Go version > > On Fri, Jan 31, 2020 at 6:19 AM Antoine Pitrou wrote: >> >> >> On Ubuntu 18.04, the source verification is successful until the go >> step, which fails: >> https://gist.github.com/pitrou/7e089ac146197b1141585c271cb39866 >> >> Side note: the JS verification step should avoid spamming the terminal >> with tons of useless information. >> >> Regards >> >> Antoine. >> >> >> Le 31/01/2020 à 04:13, Krisztián Szűcs a écrit : >>> Hi, >>> >>> I would like to propose the following release candidate (RC2) of Apache >>> Arrow version 0.16.0. This is a release consisting of 728 >>> resolved JIRA issues[1]. >>> >>> This release candidate is based on commit: >>> 729a7689fd87572e6a14ad36f19cd579a8b8d9c5 [2] >>> >>> The source release rc2 is hosted at [3]. >>> The binary artifacts are hosted at [4][5][6][7]. >>> The changelog is located at [8]. >>> >>> Please download, verify checksums and signatures, run the unit tests, >>> and vote on the release. See [9] for how to validate a release candidate. >>> >>> The vote will be open for at least 72 hours.
[jira] [Created] (ARROW-7733) [Developer] Install locally a new enough version of Go for release verification script
Wes McKinney created ARROW-7733: --- Summary: [Developer] Install locally a new enough version of Go for release verification script Key: ARROW-7733 URL: https://issues.apache.org/jira/browse/ARROW-7733 Project: Apache Arrow Issue Type: Improvement Components: Developer Tools Reporter: Wes McKinney Fix For: 1.0.0 This will ensure that if a developer has a too-old version of Go installed on their system that the release verification will still work -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: [VOTE] Release Apache Arrow 0.16.0 - RC2
OK, yes that is too old. We should install a sufficiently new version of Go as part of the release verification script to avoid this class of failure https://issues.apache.org/jira/browse/ARROW-7733 If you're interested in installing a newer version of Go on Ubuntu 18.04, here is how I do it when I provision my personal machines https://github.com/wesm/dev-toolchain/blob/master/ubuntu-bootstrap.sh#L254 On Fri, Jan 31, 2020 at 1:12 PM Antoine Pitrou wrote: > > > It's the version packaged in Ubuntu 18.04. > > $ go version > go version go1.10.4 linux/amd64 > > Regards > > Antoine. > > > Le 31/01/2020 à 20:06, Wes McKinney a écrit : > > hi Antoine -- which version of Go are you using? Some of these issues > > are resolved by upgrading the Go version > > > > On Fri, Jan 31, 2020 at 6:19 AM Antoine Pitrou wrote: > >> > >> > >> On Ubuntu 18.04, the source verification is successful until the go > >> step, which fails: > >> https://gist.github.com/pitrou/7e089ac146197b1141585c271cb39866 > >> > >> Side note: the JS verification step should avoid spamming the terminal > >> with tons of useless information. > >> > >> Regards > >> > >> Antoine. > >> > >> > >> Le 31/01/2020 à 04:13, Krisztián Szűcs a écrit : > >>> Hi, > >>> > >>> I would like to propose the following release candidate (RC2) of Apache > >>> Arrow version 0.16.0. This is a release consisting of 728 > >>> resolved JIRA issues[1]. > >>> > >>> This release candidate is based on commit: > >>> 729a7689fd87572e6a14ad36f19cd579a8b8d9c5 [2] > >>> > >>> The source release rc2 is hosted at [3]. > >>> The binary artifacts are hosted at [4][5][6][7]. > >>> The changelog is located at [8]. > >>> > >>> Please download, verify checksums and signatures, run the unit tests, > >>> and vote on the release. See [9] for how to validate a release candidate. > >>> > >>> The vote will be open for at least 72 hours.
[jira] [Created] (ARROW-7734) [C++] Segfault when comparing status with and without detail
David Li created ARROW-7734: --- Summary: [C++] Segfault when comparing status with and without detail Key: ARROW-7734 URL: https://issues.apache.org/jira/browse/ARROW-7734 Project: Apache Arrow Issue Type: Bug Components: C++ Affects Versions: 0.16.0 Reporter: David Li Assignee: David Li I noticed this while working on Flight integration tests. The equality operator for Status doesn't check whether the status detail is nullptr before dereferencing it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
